Tostore #

English | 简体中文 | 日本語 | 한국어 | Español | Português (Brasil) | Русский | Deutsch | Français | Italiano | Türkçe

Tostore is a cross-platform distributed architecture database engine deeply integrated into your project. Its neural network-inspired data processing model implements brain-like data management. Multi-partition parallelism mechanisms and node interconnection topology create an intelligent data network, while parallel processing with Isolate fully utilizes multi-core capabilities. With various distributed primary key algorithms and unlimited node extension, it can serve as a data layer for distributed computing and large-scale data training infrastructures, enabling seamless data flow from edge devices to cloud servers. Features like precise schema change detection, intelligent migration, ChaCha20Poly1305 encryption, and multi-space architecture perfectly support various application scenarios, from mobile apps to server-side systems.

Why Choose Tostore? #

1. Partition Parallel Processing vs. Single File Storage #

2. Dart Parallelism vs. Traditional Script Languages #

3. Embedded Integration vs. Independent Data Stores #

4. Precise Schema Change Detection vs. Manual Migration Management #

5. Multi-Space Architecture vs. Single Storage Space #

6. Cross-Platform Support vs. Platform Limitations #

7. Distributed Primary Key Algorithms vs. Traditional Auto-Increment IDs #

8. Streaming Data Processing vs. Batch Loading #

9. Intelligent Caching Strategies vs. Traditional Caching #

10. Serverless Architecture vs. Traditional Server Dependency #

11. Intelligent Resource Optimization vs. Fixed Resource Allocation #

Technical Highlights #

• 🌐 Transparent Cross-Platform Support:

  • Consistent experience across Web, Linux, Windows, Mobile, Mac platforms
  • Unified API interface, hassle-free cross-platform data synchronization
  • Automatic adaptation to various cross-platform storage backends (IndexedDB, file systems, etc.)
  • Seamless data flow from edge computing to cloud

• 🧠 Neural Network-Inspired Distributed Architecture:

  • Neural network-like topology of interconnected nodes
  • Efficient data partitioning mechanism for distributed processing
  • Intelligent dynamic workload balancing
  • Support for unlimited node extension, easy construction of complex data networks

• ⚡ Ultimate Parallel Processing Capabilities:

  • Truly parallel read/write with Isolates, full multi-core CPU utilization
  • Multi-node compute network cooperating for multiplied efficiency of multiple tasks
  • Resource-aware distributed processing framework, automatic performance optimization
  • Streaming query interface optimized for processing massive datasets

• 🔑 Various Distributed Primary Key Algorithms:

  • Sequential increment algorithm - freely adjustable random step length
  • Timestamp-based algorithm - ideal for high-performance parallel execution scenarios
  • Date-prefixed algorithm - suitable for data with time range indication
  • Short code algorithm - concise unique identifiers

• 🔄 Intelligent Schema Migration:

  • Precise identification of table/field renaming behaviors
  • Automatic data updating and migration during schema changes
  • Zero-downtime upgrades, no impact on business operations
  • Safe migration strategies preventing data loss

• 🛡️ Enterprise-Level Security:

  • ChaCha20Poly1305 encryption algorithm to protect sensitive data
  • End-to-end encryption, ensuring security of stored and transmitted data
  • Fine-grained data access control

• 🚀 Intelligent Caching and Search Performance:

  • Multi-level intelligent caching mechanism for efficient data retrieval
  • Startup caching dramatically improving app launch speed
  • Storage engine deeply integrated with cache, no additional synchronization code needed
  • Adaptive scaling, maintaining stable performance even with growing data

• 🔄 Innovative Workflows:

  • Multi-space data isolation, perfect support for multi-tenant, multi-user scenarios
  • Intelligent workload assignment between compute nodes
  • Provides robust database for large-scale data training and analysis
  • Automatic data storage, intelligent updates and insertions

• 💼 Developer Experience is Priority:

  • Detailed bilingual documentation and code comments (Chinese and English)
  • Rich debugging information and performance metrics
  • Built-in data validation and corruption recovery capabilities
  • Zero-configuration out-of-the-box, quick start

Quick Start #

Basic usage:

// Database initialization
final db = ToStore();
await db.initialize(); // Optional, ensures database initialization is completed before operations

// Data insertion
await db.insert('users', {
  'username': 'John',
  'email': 'john@example.com',
});

// Data updating
await db.update('users', {
  'age': 31,
}).where('id', '=', 1);

// Data deletion
await db.delete('users').where('id', '!=', 1);

// Support for complex chained queries
final users = await db.query('users')
    .where('age', '>', 20)
    .where('name', 'like', '%John%')
    .or()
    .whereIn('id', [1, 2, 3])
    .orderByDesc('age')
    .limit(10);

// Automatic data storage, update if exists, insert if not
await db.upsert('users', {'name': 'John','email': 'john@example.com'})
  .where('email', '=', 'john@example.com');
// Or
await db.upsert('users', {
  'id': 1,
  'name': 'John',
  'email': 'john@example.com'
});

// Efficient record counting
final count = await db.query('users').count();

// Processing massive data using streaming queries
db.streamQuery('users')
  .where('email', 'like', '%@example.com')
  .listen((userData) {
    // Process each record as needed, avoiding memory pressure
    print('Processing user: ${userData['username']}');
  });

// Set global key-value pairs
await db.setValue('isAgreementPrivacy', true, isGlobal: true);

// Get data from global key-value pairs
final isAgreementPrivacy = await db.getValue('isAgreementPrivacy', isGlobal: true);

Mobile App Example #

// Table structure definition suitable for frequent startup scenarios like mobile apps, precise detection of table structure changes, automatic data upgrade and migration
final db = ToStore(
  schemas: [
    const TableSchema(
      name: 'users', // Table name
      tableId: "users",  // Unique table identifier, optional, used for 100% identification of renaming requirements, even without can achieve >98% accuracy rate
      primaryKeyConfig: PrimaryKeyConfig(
        name: 'id', // Primary key
      ),
      fields: [ // Field definition, does not include primary key
        FieldSchema(name: 'username', type: DataType.text, nullable: false, unique: true),
        FieldSchema(name: 'email', type: DataType.text, nullable: false, unique: true),
        FieldSchema(name: 'last_login', type: DataType.datetime),
      ],
      indexes: [ // Index definition
        IndexSchema(fields: ['username']),
        IndexSchema(fields: ['email']),
      ],
    ),
  ],
);

// Switch to user space - data isolation
await db.switchSpace(spaceName: 'user_123');

Backend Server Example #

await db.createTables([
      const TableSchema(
        name: 'users', // Table name
        primaryKeyConfig: PrimaryKeyConfig(
          name: 'id', // Primary key
          type: PrimaryKeyType.timestampBased,  // Primary key type
        ),
        fields: [
          // Field definition, does not include primary key
          FieldSchema(
              name: 'username',
              type: DataType.text,
              nullable: false,
              unique: true),
          FieldSchema(name: 'vector_data', type: DataType.blob),  // Vector data storage
          // Other fields...
        ],
        indexes: [
          // Index definition
          IndexSchema(fields: ['username']),
          IndexSchema(fields: ['email']),
        ],
      ),
      // Other tables...
]);


// Table structure update
final taskId = await db.updateSchema('users')
    .renameTable('newTableName')  // Rename table
    .modifyField('username',minLength: 5,maxLength: 20,unique: true)  // Modify field properties
    .renameField('oldName', 'newName')  // Rename field
    .removeField('fieldName')  // Remove field
    .addField('name', type: DataType.text)  // Add field
    .removeIndex(fields: ['age'])  // Remove index
    .setPrimaryKeyConfig(  // Set primary key configuration
      const PrimaryKeyConfig(type: PrimaryKeyType.shortCode)
    );
    
// Query migration task status
final status = await db.queryMigrationTaskStatus(taskId);  
print('Migration progress: ${status?.progressPercentage}%');

Distributed Architecture #

// Distributed node configuration
final db = ToStore(
    config: DataStoreConfig(
        distributedNodeConfig: const DistributedNodeConfig(
            enableDistributed: true,  // Enable distributed mode
            clusterId: 1,  // Cluster membership configuration
            centralServerUrl: 'http://127.0.0.1:8080',
            accessToken: 'b7628a4f9b4d269b98649129'))
);

// Batch insertion of vector data
await db.batchInsert('vector', [
  {'vector_name': 'face_2365', 'timestamp': DateTime.now()},
  {'vector_name': 'face_2366', 'timestamp': DateTime.now()},
  // ... Thousands of records
]);

// Streaming processing of large datasets for analysis
await for (final record in db.streamQuery('vector')
    .where('vector_name', '=', 'face_2366')
    .where('timestamp', '>=', DateTime.now().subtract(Duration(days: 30)))
    .stream) {
  // Streaming interface supports large-scale feature extraction and transformation
  print(record);
}

Primary Key Examples #

Various primary key algorithms, all supporting distributed generation, not recommended to generate primary keys yourself to avoid the impact of unordered primary keys on search capabilities. Sequential primary key PrimaryKeyType.sequential: 238978991 Timestamp-based primary key PrimaryKeyType.timestampBased: 1306866018836946 Date-prefixed primary key PrimaryKeyType.datePrefixed: 20250530182215887631 Short code primary key PrimaryKeyType.shortCode: 9eXrF0qeXZ

// Sequential primary key PrimaryKeyType.sequential
// When distributed system is enabled, central server allocates ranges that nodes generate themselves, compact and easy to remember, suitable for user IDs and attractive numbers
await db.createTables([
      const TableSchema(
        name: 'users',
        primaryKeyConfig: PrimaryKeyConfig(
          type: PrimaryKeyType.sequential,  // Sequential primary key type
          sequentialConfig:  SequentialIdConfig(
              initialValue: 10000, // Auto-increment starting value
              increment: 50,  // Increment step
              useRandomIncrement: true,  // Use random step to avoid revealing business volume
            ),
        ),
        // Field and index definition...
        fields: []
      ),
      // Other tables...
 ]);

Security Configuration #

// Table and field renaming - automatic recognition and data preservation
final db = ToStore(
      config: DataStoreConfig(
        enableEncoding: true, // Enable secure encoding for table data
        encodingKey: 'YouEncodingKey', // Encoding key, can be adjusted arbitrarily
        encryptionKey: 'YouEncryptionKey', // Encryption key, note: changing this will prevent decoding of old data
      ),
    );

Performance Tests #

Tostore 2.0 implements linear performance scalability, fundamental changes in parallel processing, partitioning mechanisms, and distributed architecture have significantly improved data search capabilities, providing millisecond response times even with massive data growth. For processing large datasets, the streaming query interface can process massive data volumes without exhausting memory resources.

Future Plans #

Tostore is developing support for high-dimensional vectors to adapt to multimodal data processing and semantic search.

Our goal is not to create a database; Tostore is simply a component extracted from the Toway framework for your consideration. If you are developing mobile applications, we recommend using the Toway framework with its integrated ecosystem, which covers the full-stack solution for Flutter application development. With Toway, you won't need to touch the underlying database, all query, loading, storage, caching, and data display operations will be automatically performed by the framework. For more information about the Toway framework, please visit the Toway repository.

Documentation #

Visit our Wiki for detailed documentation.

Support and Feedback #

• Submit issues: GitHub Issues
• Join the discussion: GitHub Discussions
• Contribute code: Contributing Guide

License #

This project is licensed under the MIT License - see the LICENSE file for details.

If you find Tostore useful, please give us a ⭐️