Using CONCAT and string manipulation functions: SELECT CONCAT(‘{ “column1”: “‘, column1, ‘”, “column2”: “‘, column2, ‘”, “column3”: “‘, column3, ‘” }’) AS json_data FROM your_table; Using a custom UDF (User-Defined Function): If the built-in...
Top Apache Spark Interview Questions and Answers/ Top Data Engineer Interview Questions and Answers WAQ to find 2nd highest salary using window functions WAQ to find employees having salary higher than their manager Partitioning...
Apache Pulsar and Apache Kafka are both open-source distributed systems that are designed to handle high-performance, low-latency streaming data. However, they have different architectures and use cases. Apache Pulsar is a distributed pub-sub messaging...
Apache Kafka can be integrated with REST (Representational State Transfer) APIs to enable communication between Kafka and other systems that use REST. Different ways to achieve this, but one popular approach is to use...
Apache Kafka is an open-source, distributed streaming platform used to build real-time data pipelines and streaming applications. It was developed by LinkedIn and later donated to the Apache Software Foundation in 2011. Kafka is...
Different type of NOSQL Databases Type of NoSQL Database Examples Key-value Redis, Riak Document MongoDB, Couchbase Column-family Apache Cassandra, Hbase Graph Neo4j, Amazon Neptune Time-series InfluxDB, OpenTSDB Object db4o, ZopeDB Search Engine Elasticsearch, Solr...
NoSQL and SQL (Structured Query Language) are two different types of databases. SQL databases, also known as relational databases, store data in a structured format in tables, with rows and columns. They use a...
Here are some examples of transformations in PySpark: Map map(func) – Applies the given function to each element of the dataset and returns a new dataset with the results. For example: from pyspark.sql.functions import...
In Apache Spark, transformation and action are two types of operations that can be performed on RDDs (Resilient Distributed Datasets) or DataFrames/Datasets. Transformations are operations that are performed on an RDD or DataFrame/Dataset that...
ORC (Optimized Row Columnar) is a file format that is optimized for reading and writing large datasets in a columnar format. It was developed by the Apache Hive project and is widely used in...