Skip to main content

Sharding

Sharding is a data management strategy used to enhance the performance and scalability of databases by dividing a large dataset into smaller, more manageable pieces known as "shards." This method allows for the distribution of data across multiple servers, enabling systems to handle increased loads and larger datasets more efficiently.

Overview of Sharding​

Sharding involves splitting a database into smaller segments, which can be stored on different database servers or instances. Each shard contains a unique subset of the data, ensuring that all shards together represent the entire dataset. This approach is particularly beneficial for applications with high transaction volumes or extensive data storage needs, as it allows for horizontal scaling—adding more machines to share the load—rather than vertical scaling, which involves upgrading a single machine's resources

Types of Sharding​

Sharding can be categorized into two main types: horizontal sharding and vertical sharding.

Horizontal Sharding​

Horizontal sharding, also known as "data partitioning," involves dividing a database table into smaller tables (shards) based on rows. Each shard contains a unique set of rows, but all shards maintain the same schema. This method is particularly effective when queries often target specific subsets of rows, such as user data segmented by geographical location or date ranges. For instance, a social media application might shard user data by region, allowing users from North America to be stored on one server and users from Europe on another

Advantages of Horizontal Sharding​

Query Efficiency: Queries that filter data based on specific criteria can be directed to the relevant shard, minimizing the amount of data scanned and improving response times. Load Balancing: By distributing users and their data across multiple shards, the system can balance traffic and reduce bottlenecks during peak usage times

Vertical Sharding​

Vertical sharding divides a database table based on columns rather than rows. In this case, different columns are stored in separate shards, allowing for the isolation of frequently accessed data. For example, a wide table with many columns might be split into two tables: one containing user names and another containing user addresses. This method is particularly useful when different queries access different subsets of columns, optimizing performance by reducing the amount of data processed for each query.

Advantages of Vertical Sharding​

Optimized Data Access: Queries that require only a subset of columns can be executed more efficiently, as they access smaller, more focused shards. Reduced Complexity for Analytics: In environments where analytical workloads are common, vertical sharding can simplify data retrieval and improve performance by isolating the most relevant data

Sharding at Api Verse​

At Api verse, we plan to use the sharding mechanism in a novel way to increase the decentralization and availability of the ability to host services by those with even small server resources.

Horizontal sharding​

Horizontal sharding in the case of Api Verse allows each data provider to host only data from a selected list of networks. For example, only those of the highest interest and queried most frequently. It can also look from the point of view of the resources it has at its disposal and focus on networks that have fewer transactions and therefore require less disk and memory resources.

Example​

Let's look at the example of a person who has limited hardware resources. It can't afford to put data from more than 100 networks on its infrastructure, because that would take up more than 1 terabyte of data and require huge storage resources. Therefore, the data provider can choose a single network or a collection of selected networks and deliver data only from them. Depending on its strategy, it can choose the most heavily trafficked networks to expect the highest rewards. Alternatively, he can focus on networks with a low number of transactions so that hosting them does not require a large amount of memory.

Vertical sharding​

Vertical sharding in the case of Api Verse allows each data provider to host only part of selected network. For example, only those of the highest interest and queried most frequently. It can also look from the point of view of the resources it has at its disposal and focus on networks that have fewer transactions and therefore require less disk and memory resources.

This further reduces hardware resource requirements. Instead of storing the entire data associated with a selected blockchain network, an operator can focus on storing only a selected area of data - for example, containing the state of cryptocurrency wallets on any block. If the data provider targets the right areas from the selected networks, it can push hard to increase decentralization and network uptime and, in parallel, reap significant rewards over the infrastructure it has to host.

Data providing economy​

The novel approach to data provisioning in the VERSE API means that the data provider does not have to be just a passive participant whose infrastructure performs the calculations. Through the possibility of vertical and horizontal sharding, it can search for opportunities in the market by skillfully finding areas where the rewards obtained relative to the server infrastructure are highest. This will also cause profit-maximizing oriented individuals to dynamically manage their infrastructure and jump from area to area.