这是一个为期一学期的分布式系统编程项目，要求学生用C语言实现一个高性能的键值存储系统。项目要求采用客户端-服务器架构，支持多并发连接，包含网络协议设计、存储引擎实现、并发控制等核心模块。系统需具备持久化存储、故障恢复、性能监控等功能，并通过严格的测试验证。最终提交需包含完整代码、技术文档和测试报告，评估标准包括正确性、性能、代码质量等。项目分13周完成，从设计到优化逐步推进，旨在培养学生系统编程和分布式计算的实践能力。
Project Overview
This semester-long project challenges you to design and implement a distributed key-value store in C that demonstrates mastery of systems programming concepts. The assignment focuses on building a robust, high-performance storage system while addressing real-world challenges in distributed computing.
System Architecture Requirements
The key-value store must employ a client-server architecture with support for multiple concurrent clients. Your implementation should feature a modular design separating the networking layer, storage engine, and consensus protocol. The server component needs to handle at least 100 simultaneous connections while maintaining consistent performance.
Core Storage Engine Implementation
At the heart of your system lies the storage engine responsible for data persistence. You must implement both an in-memory hash table and a disk-based log-structured storage component. The engine should support configurable eviction policies including LRU and LFU, with careful consideration given to memory management strategies.
Network Protocol Design
Develop a custom binary protocol for client-server communication that includes support for basic CRUD operations, batch processing, and atomic transactions. The protocol must implement proper framing, error detection, and request pipelining. Special attention should be paid to efficient serialization formats and minimizing network overhead.
Concurrency and Synchronization
The system must employ efficient synchronization primitives to handle concurrent access to shared data structures. Your implementation should demonstrate understanding of various locking strategies, lock-free algorithms, and appropriate use of thread-local storage. Include mechanisms to prevent common concurrency issues like deadlocks and priority inversion.
Persistence and Recovery
Implement a write-ahead log (WAL) to ensure durability and support crash recovery. The persistence layer should handle fsync operations strategically to balance durability requirements with performance. Your solution must include tools for log inspection and database repair in case of corruption.
Performance Optimization
Instrument your code to collect detailed performance metrics including operation latency, throughput, and resource utilization. Implement and compare multiple caching strategies, evaluating their impact on different workload patterns. The system should include adaptive mechanisms that tune parameters based on observed behavior.
Testing and Verification Framework
Develop a comprehensive testing harness that validates both functional correctness and performance characteristics. Your test suite should include:
- Unit tests for core data structures
- Integration tests for distributed operations
- Fault injection tests for resilience verification
- Long-running stress tests for stability assessment
Security Considerations
Implement proper authentication and authorization mechanisms for client connections. Include safeguards against common vulnerabilities like buffer overflows and injection attacks. Your design should incorporate principles of least privilege and defense in depth.
Monitoring and Diagnostics
Build an administrative interface that exposes system metrics and operational status. Implement logging at multiple verbosity levels with careful attention to performance impact. Include tools for runtime configuration tuning and problem diagnosis.
Documentation Requirements
Produce detailed technical documentation covering:
- Architectural decisions and tradeoffs
- Protocol specifications
- Performance characteristics
- Operational guidelines
- Known limitations and future improvements
The documentation should be thorough enough to enable another engineer to maintain and extend your system.
Evaluation Criteria
Your submission will be assessed on:
- Correctness and robustness
- Architectural coherence
- Performance characteristics
- Code organization and quality
- Depth of analysis in documentation
- Completeness of test coverage
Implementation Advice
Adopt an iterative development approach, focusing first on core functionality before optimizing. Use version control effectively to manage your codebase. Start performance benchmarking early to identify bottlenecks. Consider using a project management methodology to track progress.
Suggested Timeline
Week 1-3: Design and prototype core components
Week 4-6: Implement basic functionality
Week 7-9: Add advanced features and optimize
Week 10-12: Testing, documentation, and final tuning
Week 13: System demonstration and review
Technical Constraints
The implementation must:
- Use only standard C libraries (no external dependencies)
- Compile with strict warning flags enabled
- Run on Linux systems
- Support x86_64 architecture
- Maintain consistent performance under load
Extension Opportunities
For those seeking additional challenges, consider:
- Adding TLS support for secure communication
- Implementing data compression
- Developing a replication protocol
- Creating a management CLI
- Building a benchmarking tool
The project provides an excellent opportunity to demonstrate your understanding of complex systems programming concepts while creating a substantial piece of software. Approach it methodically and don’t hesitate to seek clarification on requirements.