Skip to content

Commit

Permalink
merge
Browse files Browse the repository at this point in the history
  • Loading branch information
@emanuelchristo
emanuelchristo committed Apr 27, 2024
2 parents 0f6b6d6 + 667b4f7 commit c6fec8b
Show file tree
Hide file tree
Showing 36 changed files with 1,796 additions and 1,566 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/deploy.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
- name: Install dependencies
run: yarn install # or pnpm install / yarn install / bun install
- name: Build with VitePress
run: yarn docs:build # or pnpm docs:build / yarn docs:build / bun run docs:build
run: yarn build # or pnpm docs:build / yarn docs:build / bun run docs:build
- name: Upload artifact
uses: actions/upload-pages-artifact@v3
with:
Expand Down
84 changes: 48 additions & 36 deletions docs/.vitepress/config.mjs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,21 +80,25 @@ export default defineConfig({
link: '/roadmap/phase-0/stage-0.md',
},
{
text: 'Stage 1: TCP server',
text: 'Stage 1: TCP Server',
link: '/roadmap/phase-0/stage-1',
},
{
text: 'Stage 2: TCP client',
text: 'Stage 2: TCP Client',
link: '/roadmap/phase-0/stage-2',
},
{
text: 'Stage 3: Epoll',
text: 'Stage 3: Linux epoll',
link: '/roadmap/phase-0/stage-3',
},
{
text: 'Stage 4: TCP proxy',
text: 'Stage 4: UDP Multi-threading',
link: '/roadmap/phase-0/stage-4',
},
{
text: 'Stage 5: TCP Proxy',
link: '/roadmap/phase-0/stage-5',
},
],
},
{
Expand All @@ -106,25 +110,29 @@ export default defineConfig({
link: '/roadmap/phase-1/',
},
{
text: 'Stage 5: Server & Client module',
link: '/roadmap/phase-1/stage-5',
},
{
text: 'Stage 6: Core & Loop module',
text: 'Stage 6: Listener & Connection Modules',
link: '/roadmap/phase-1/stage-6',
},
{
text: 'Stage 7: TCP module',
text: 'Stage 7: Core & Loop Modules',
link: '/roadmap/phase-1/stage-7',
},
{
text: 'Stage 8: Upstream module',
text: 'Stage 8: Pipe Module',
link: '/roadmap/phase-1/stage-8',
},
{
text: 'Stage 9: File module',
text: 'Stage 9: Upstream Module',
link: '/roadmap/phase-1/stage-9',
},
{
text: 'Stage 10: File module',
link: '/roadmap/phase-1/stage-10',
},
{
text: 'Stage 11: Session Module',
link: '/roadmap/phase-1/stage-11',
},
],
},
{
Expand All @@ -136,20 +144,20 @@ export default defineConfig({
link: '/roadmap/phase-2/',
},
{
text: 'Stage 10: HTTP parser',
link: '/roadmap/phase-2/stage-10',
text: 'Stage 12: HTTP Request Module',
link: '/roadmap/phase-2/stage-12',
},
{
text: 'Stage 11: HTTP req & HTTP res',
link: '/roadmap/phase-2/stage-11',
text: 'Stage 13: HTTP Response Module',
link: '/roadmap/phase-2/stage-13',
},
{
text: 'Stage 12: Config & Session module',
link: '/roadmap/phase-2/stage-12',
text: 'Stage 14: Config Module',
link: '/roadmap/phase-2/stage-14',
},
{
text: 'Stage 13: HTTP Spec',
link: '/roadmap/phase-2/stage-13',
text: 'Stage 15: HTTP Specification',
link: '/roadmap/phase-2/stage-15',
},
],
},
Expand All @@ -162,24 +170,24 @@ export default defineConfig({
link: '/roadmap/phase-3/',
},
{
text: 'Stage 14: IP whitelist & blacklist',
link: '/roadmap/phase-3/stage-14',
text: 'Stage 16: Directory Browsing',
link: '/roadmap/phase-3/stage-16',
},
{
text: 'Stage 15: Directory browsing',
link: '/roadmap/phase-3/stage-15',
text: 'Stage 17: IP Whitelist/Blacklist',
link: '/roadmap/phase-3/stage-17',
},
{
text: 'Stage 16: Compression',
link: '/roadmap/phase-3/stage-16',
text: 'Stage 18: Gzip Compression',
link: '/roadmap/phase-3/stage-18',
},
{
text: 'Stage 17: Load balancing',
link: '/roadmap/phase-3/stage-17',
text: 'Stage 19: Load Balancing',
link: '/roadmap/phase-3/stage-19',
},
{
text: 'Stage 18: Rate limiting & timeouts',
link: '/roadmap/phase-3/stage-18',
text: 'Stage 20: Request timeouts',
link: '/roadmap/phase-3/stage-20',
},
],
},
Expand All @@ -192,16 +200,20 @@ export default defineConfig({
link: '/roadmap/phase-4/',
},
{
text: 'Stage 19: TLS',
link: '/roadmap/phase-4/stage-19',
text: 'Stage 21: Metrics',
link: '/roadmap/phase-4/stage-21',
},
{
text: 'Stage 20: Caching',
link: '/roadmap/phase-4/stage-20',
text: 'Stage 22: Multiprocess',
link: '/roadmap/phase-4/stage-22',
},
{
text: 'Stage 21: Multiprocess Architecture',
link: '/roadmap/phase-4/stage-21',
text: 'Stage 23: Caching',
link: '/roadmap/phase-4/stage-23',
},
{
text: 'Stage 24: Transport Layer Security (TLS)',
link: '/roadmap/phase-4/stage-24',
},
],
},
Expand Down
8 changes: 2 additions & 6 deletions docs/guides/references/vec.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

A type-safe dynamic array implementation for C.

https://github.com/rxi/vec
This library is a copy of the work at https://github.com/rxi/vec and is Copyrighted by [rxi](https://rxi.github.io/) (c) 2014.
[MIT License](https://github.com/rxi/vec/blob/master/LICENSE)

## Installation

Expand Down Expand Up @@ -517,8 +518,3 @@ vec_foreach_ptr(&v, val, i) {

Iterates the value pointers of the vector from last to first. See
`vec_foreach_ptr()`

## License

This library is free software; you can redistribute it and/or modify it under
the terms of the MIT license.
20 changes: 10 additions & 10 deletions docs/guides/resources/architecture.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,15 @@ A detailed view of the _structs_ and _functions_ present in each module will be

### `xps_core`

The `xps_core` module serves as the container to which [instances](<https://en.wikipedia.org/wiki/Instance_(computer_science)>) of all other modules are attached. It can be thought of as an instance of eXpServer.
The `xps_core` module serves as the container to which [instances](<https://en.wikipedia.org/wiki/Instance_(computer_science)>) of all other modules are attached. It can be thought of as an instance of eXpServer itself.

### `xps_loop`

The `xps_loop` module contains the [event loop](https://en.wikipedia.org/wiki/Event_loop). Event loop is the engine that drives eXpServer. It is implemented using Linux [epoll](https://en.wikipedia.org/wiki/Epoll). TCP sockets are attached to epoll to monitor for events. On receiving event notifications, the loop will handle them through [callback functions](<https://en.wikipedia.org/wiki/Callback_(computer_programming)>). Another responsibility of the loop is to drive the `xps_pipe`, through which the bytes flow from one module to another (`xps_pipe` will be explained subsequently).

### `xps_config`

The `xps_config` module is responsible for reading and parsing the configuration file, the path to which is provided as a [command line argument](https://en.wikipedia.org/wiki/Command-line_interface#Arguments). The configuration file is written and parsed using [Lua](<https://en.wikipedia.org/wiki/Lua_(programming_language)>) into a configuration _struct_ and stored in the `xps_core` instance.
The `xps_config` module is responsible for reading and parsing the configuration file, the path to which is provided as a [command line argument](https://en.wikipedia.org/wiki/Command-line_interface#Arguments). The configuration file will be a JSON file with a specific structure. It is parsed using [parson](https://github.com/kgabis/parson), a 3rd party JSON parser. After parsing, it is stored in the `xps_core` instance as a configuration _struct_.

### `xps_listener`

Expand All @@ -71,11 +71,11 @@ The `xps_connection` module creates instances for TCP connections, be it a clien

### `xps_session`

An `xps_sesion` instance is created in the previously mentioned `listener_connection_handler()` function at the same time a new client connection is accepted. The session instance is the orchestrator that handles the client requests. It will parse the incoming bytes from the client TCP connection using the `xps_http` module. Based on the parsed HTTP request, the session instance will lookup the configuration to determine whether it should serve a file or reverse proxy the request. On deciding, an `xps_file` instance or an `xps_upstream` instance is created and attached using `xps_pipe` to the session instance. Then the bytes flow between the _client_ and corresponding _file_ or _upstream_ through the _session_ instance.
An `xps_sesion` instance is created in the previously mentioned `listener_connection_handler()` function at the same time a new client connection is accepted. The session instance is the orchestrator that handles the client requests. It will parse the incoming bytes from the client TCP connection using the `xps_http` module. Based on the parsed HTTP request, the session instance will lookup the configuration to determine whether it should serve a file or reverse proxy the request. On deciding, an `xps_file` instance or an `xps_upstream` instance is created and attached using `xps_pipe` to the session instance. Then the bytes flow between the _client_ and corresponding _file_ or _upstream server_ through the _session_ instance.

### `xps_http`

The `xps_http` module contains 2 _struct types_: `xps_http_req` and `xps_http_res` . `xps_http_req` takes in a [data buffer](https://en.wikipedia.org/wiki/Data_buffer) from the client and parses it according to the [HTTP specification](https://www.rfc-editor.org/rfc/rfc9110) to create the request instance. `xps_http_res` takes in a [HTTP response code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) and sets up a response instance with appropriate [HTTP headers](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields). These _types_ also come with a [serialize](https://en.wikipedia.org/wiki/Serialization) function which will convert the _struct_ to a _buffer_ that can be transmitted to the _connection_.
The `xps_http` module contains 2 _struct types_: `xps_http_req` and `xps_http_res` . `xps_http_req` takes in a data buffer which is essentially a char array allocated using `malloc()` from the client and parses it according to the [HTTP specification](https://www.rfc-editor.org/rfc/rfc9110) to create the request instance. `xps_http_res` takes in a [HTTP response code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) and sets up a response instance with appropriate [HTTP headers](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields). These _types_ also come with a [serialize](https://en.wikipedia.org/wiki/Serialization) function which will convert the _struct_ to a _buffer_ that can be transmitted to the _connection_.

### `xps_file`

Expand All @@ -91,9 +91,9 @@ The `xps_pipe` module serves as a link between various nodes, such as `xps_conne

Read more about _pipes_ below.

## Pipes
## xps_pipes

Pipes in eXpServer are the links that allow uni-directional [controlled flow](<https://en.wikipedia.org/wiki/Flow_control_(data)>) of bytes from one node to another. A pipe is an instance of `xps_pipe_t` type and is attached to a source instance of type `xps_pipe_source_t` on one end and a sink instance of type `xps_pipe_sink_t` on the other.
Pipes in eXpServer are the links that allow uni-directional [controlled flow](<https://en.wikipedia.org/wiki/Flow_control_(data)>) of bytes from one node to another. An `xps_pipe` is an instance of `xps_pipe_t` type and is attached to a source instance of type `xps_pipe_source_t` on one end and a sink instance of type `xps_pipe_sink_t` on the other.

Let us take a look at an example

Expand All @@ -109,8 +109,8 @@ Now let us look at how this works
- When a pipe is created, it is added to a list of pipes present in the core.
- With each iteration of the event loop, a `handle_pipes()` function is invoked.
- The `handle_pipes()` function will iterate over all the pipes in core and checks for the following conditions
- If the pipe is writable ie. amount of bytes in pipe buffer is less than the pipe buffer threshold AND if the source is ready ie. source has some data to write to the pipe, then a callback function - `handler_cb()` on the source is invoked which will proceed to write to the pipe.
- If the pipe is readable ie. amount of bytes in pipe buffer is greater than 0 AND if sink is ready ie. sink is available to read some data from the pipe, then a callback function - `handler_cb()` on the sink is invoked which will proceed to read from the pipe.
- If the pipe is writable i.e. amount of bytes in pipe buffer is less than the pipe buffer threshold AND if the source is ready i.e. source has some data to write to the pipe, then a callback function - `handler_cb()` on the source is invoked which will proceed to write to the pipe.
- If the pipe is readable i.e. amount of bytes in pipe buffer is greater than 0 AND if sink is ready i.e. sink is available to read some data from the pipe, then a callback function - `handler_cb()` on the sink is invoked which will proceed to read from the pipe.
- If the pipe has no source and sink attached to it, then the pipe is destroyed
- If the pipe has a source and no sink, a callback function - `close_cb()` on the source will be invoked which will notify the source that there is no sink attached to it and can destroy its instance if it wants to.
- If the pipe has a sink and no source AND the pipe is not readable, then a callback function - `close_cb()` on the sink will be invoked which will notify the sink that there is no source attached to it and there is nothing left to read from the pipe and hence can destroy its instance if it wants to.
Expand All @@ -124,10 +124,10 @@ Now let us look at how this works
In our case,

- If the source is writing to the pipe faster than the sink is reading it, then the pipe buffer will fill up and reach above the buffer threshold. If that happens, even if the source is ready, `handler_cb()` on the source will not be invoked, there by ‘blocking’ the source from writing to the pipe.
- Similarly, if the sink is reading from the pipe faster than the source is writing to it, then, when the pipe becomes empty the `handler_cb()` on the sink will not be invoked even if the sink is ready, there by ‘blocking’ the sink from reading from the pipe
- Similarly, if the sink is reading from the pipe faster than the source is writing to it, then, when the pipe becomes empty, the `handler_cb()` on the sink will not be invoked even if the sink is ready, thereby ‘blocking’ the sink from reading from the pipe

### Isolation of Logic

The problem with of having a central entity, say the event loop, read directly from the file FD and write directly to the socket FD is that the logic associated with a particular module will be spread across multiple modules. This can lead to unmaintainable code and increased complexity when extending eXpServer with additional modules.

With the pipe approach, module logic is encapsulated within them with their data endpoints being exposed in the form of sources and sinks. This means that any module with a source can be attached to any other module with sink providing a easy interface for inter module communication.
With the pipe approach, module logic is encapsulated within them with their data endpoints being exposed in the form of sources and sinks. This means that any module with a source can be attached to any other module with sink providing an easy interface for inter module communication.
39 changes: 21 additions & 18 deletions docs/roadmap/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,37 +23,40 @@ The eXpServer project comprises 22 stages, organized into 5 phases. Prior to the
- [Stage 1: TCP Server](phase-0/stage-1)
- [Stage 2: TCP Client](phase-0/stage-2)
- [Stage 3: Linux epoll](phase-0/stage-3)
- [Stage 4: TCP Proxy](phase-0/stage-4)
- [Stage 4: UDP with Multi-threading](phase-0/stage-4)
- [Stage 5: TCP Proxy](phase-0/stage-5)

### Phase 1: Building the core of eXpServer by creating reusable modules

- [Overview](phase-1/)
- [Stage 5: Server & Client Modules](phase-1/stage-5)
- [Stage 6: Core & Loop Modules](phase-1/stage-6)
- [Stage 7: TCP Module](phase-1/stage-7)
- [Stage 8: Upstream Module](phase-1/stage-8)
- [Stage 9: File Module](phase-1/stage-9)
- [Stage 6: Listener & Connection Modules](phase-1/stage-6)
- [Stage 7: Core & Loop Modules](phase-1/stage-7)
- [Stage 8: Pipe Module](phase-1/stage-8)
- [Stage 9: Upstream Module](phase-1/stage-9)
- [Stage 10: File Module](phase-1/stage-10)
- [Stage 11: Session Module](phase-1/stage-11)

### Phase 2: Implementing HTTP support

- [Overview](phase-2/)
- [Stage 10: HTTP Parser](phase-2/stage-10)
- [Stage 11: HTTP Req & Res Modules](phase-2/stage-11)
- [Stage 12: Config & Session Modules](phase-2/stage-12)
- [Stage 13: HTTP Specification](phase-2/stage-13)
- [Stage 12: HTTP Request Module](phase-2/stage-12)
- [Stage 13: HTTP Response Module](phase-2/stage-13)
- [Stage 14: Config Module](phase-2/stage-14)
- [Stage 15: HTTP Specification](phase-2/stage-15)

### Phase 3: Adding features to eXpServer

- [Overview](phase-3/)
- [Stage 14: IP Whitelist/Blacklist](phase-3/stage-14)
- [Stage 15: Directory Browsing](phase-3/stage-15)
- [Stage 16: Gzip Compression](phase-3/stage-16)
- [Stage 17: Load Balancing](phase-3/stage-17)
- [Stage 18: Rate Limiting & Timeout](phase-3/stage-18)
- [Stage 16: Directory Browsing](phase-3/stage-16)
- [Stage 17: IP Whitelist/Blacklist](phase-3/stage-17)
- [Stage 18: Gzip Compression](phase-3/stage-18)
- [Stage 19: Load Balancing](phase-3/stage-19)
- [Stage 20: Request timeouts](phase-3/stage-20)

### Phase 4: Advanced features and multiprocess architecture

- [Overview](phase-4/)
- [Stage 19: Transport Layer Security (TLS)](phase-4/stage-19)
- [Stage 20: Caching](phase-4/stage-20)
- [Stage 21: Multiprocess Architecture](phase-4/stage-21)
- [Stage 21: Metrics](phase-4/stage-22)
- [Stage 22: Multiprocess](phase-4/stage-22)
- [Stage 23: Caching](phase-4/stage-23)
- [Stage 24: Transport Layer Security (TLS)](phase-4/stage-24)
Loading

0 comments on commit c6fec8b

Please sign in to comment.