While partitioning remains a powerful technique for managing and optimizing data access, sharding could serve to be a better approach. Sharding offers enhanced parallelism, flexibility in data organization, scalability, and reduction of contention. It definitely stands out in scenarios where more granular control and customization are required. Simply put, in some scenarios sharding can be better than partitioning in BigQuery because it offers greater availability and performance.
Great news! Launchpad ETL makes use of both strategies — sharding and partitioning — as well as clustering. Launchpad ETL offers flexible features, such as selecting between a sharded destination table or a partitioned table. This powerful tool can support multiple scenarios, allowing you to achieve the best results. For example, a majority of our predefined source schemas that support batching will automatically adopt the sharding mechanism, but can be changed to a standardized date partition. Launchpad ETL makes it easy to work with extremely large sets of historical data.
