Apache Spark’s Python Data Source API simplifies data ingestion from custom sources. Now, data engineers can efficiently integrate a broader array of data sources and sinks into their Spark pipelines. "The fact that we can now build, test, and surface reusable Spark data sources

→ View original post on X — @databricks