raw Web server in C

[detergent1]

Merry Christmas!!

so! I spent the first two weeks of my vacation studying about Web servers and how to do it in C!!

honestly, I have no reason to do it. I only have static Web pages to serve. so https://neocities.org really is the way right now. I intend to explore keeping data in C and troff-like files to generate the HTML, however.

now let's say you want dynamic Web pages in C?

actually, many functionality of dynamic Web pages can be replaced by active Web pages, that is: JavaScript, or more recently: WebAssembly. AND YOU KNOW WHAT? you can compile C to JavaScript or WebAssembly using https://emscripten.org/ hahahahha

but let's say you really need dynamic Web pages

the easiest way is to setup an industry standard Web server like Apache or nginx, or something recent like H2O (notice the trend: there are tons of Web servers, but I only mention those written in C! muahahaha) then you use https://kristaps.bsd.lv/kcgi/ to easily parse and answer HTTP requests through FastCGI (or CGI if you really feel like)

and let me tell you. FastCGI and its predecessor CGI abstracts away A LOT of bullshit going under the hood.

FastCGI is a protocol between two processes: the Web server and a stream process generating HTML (or other files) on the fly. this has two advantages: multiple FastCGI streams as a form of parallelization, and replacing the FastCGI script while the server is running (understand: updating a Web site without restarting the Web server).

but let's say you want to go lower. https://nodepp.org/ is one way. it is written in C++, but it has a C interface. looking good! one executable that does everything: it is a Web server and it actually serves the Web pages. no FastCGI

but what if we want to go LOWER? for the sake of SPEED. the lowest would be so use sockets directly.

or is it?

you can go lower by keeping an userland TCP/IP stack or writing a kernel-module that bypasses the "slow" userland API, dealing directly with the Network Interface Card (NIC) and it's driver and Direct Memory Access (DMA) and kernel internals. but I'm told this unmaintainable, which defeats the whole purpose. so as others say relentlessly, we MUST trust the TCP/IP stack inside the kernels of Windows, Linux, macOS and BSD and AIX and Solaris and whatnot. because apparently the World Wide Web does not adhere strictly to TCP/IP, so a perfect implementation doesn't work in real life.

back to sockets: we have three approaches: 1) synchronous blocking API, 2) non-blocking API and 3) asynchronous API

synchronous blocking API works great if you spawn a few dozen threads. that's good for a client, but a Web server must attend millions of concurrent requests nowadays. actually, some argue that if kernels provided better more lightweight native threads, threading would and should be THE WAY.

truly asynchronous API is quite rare. surprisingly, Windows does it, and Solaris and AIX? yes, apparently Linux and BSD doesn't have true asynchronous APIs available, only emulations using... guess what: threads. an asynchronous API is ridiculously simple: "execute this callback when this operation finishes"

so Linux and BSD and macOS rely on non-blocking APIs which... kinda suck very very hard? I mean, look how simple and obvious asynchronous APIs are. now non-blocking API requires that you keep asking: is it ready? is it ready? is it ready? is it ready? is it ready? in a closed loop. and only recent APIs are okayish, in the past there were terrible attempts where you'd need to ask for readiness for EVERY open socket, or do a linear search on a list of readiness

so Linux last years added io_uring, which is very fast, but some say it is unsafe :(

the worst is: there is no standard, and most stuff is out of touch with the reality of hardware and operating systems. so every system has multiple and differing implementations.

overall, networking APIs and libraries are so braindead and out of touch and messy compared to rock-solid standardized GPU and concurrency libraries like OpenGL, DirectX, OpenMP, CUDA, MPI and pthreads.

to answer that, we have event libraries that abstract all this bullshit: libevent, libev, libuv and libhv. I listed in order of older to newest. these are asynchronous event libraries. because asynchronous is superior. so they use asynchronous APIs where available, and emulate them in systems with only shitty non-blocking APIs *stares at Linux and BSD in disappointment*. IOCP from Windows is apparently the ultimate immortal epic API.

and there is also something about RX and TX queues in NICs that impact performance and limit concurrency...

anyway. libevent is the older library, it is very portable and has many features and is used in many projects. libev is a non-blocking event library that only runs in UNIX-like systems, and is used in a few projects. libuv originated using libev to be used inside Node.js, but evolved to an asynchronous event library when adding support to Windows's IOCP, and now doesn't use libev anymore, it's scope is very constrained, it doesn't even have TLS, which can be seen as good (or bad for H2O). libhv is a chinese thing that sounds super cool, but feels like having feature creep which will probably affect reliability and security.

established Web servers have a complicated API to use raw in C: H2O, nginx and others. not worth the effort even to write a simple "Hello, World!" in text/plain.

so IF I don't use FastCGI nor node++, I'd go with libuv or libevent, and then implement HTTP support manually by stacking libraries one on top of each other. some of these libraries I discovered follows:

1. https://github.com/tatsuhiro-t/wslay
2. https://github.com/nghttp2/nghttp2
3. https://github.com/ngtcp2/nghttp3
4. https://github.com/ngtcp2/ngtcp2
5. some TLS library (with QUIC preferably, there are dozens)
6. ...
7. profit?

the POINT is: by going this low-level, I can support easily other protocols like, eeerhm, Gopher? WebSockets? and maybe add a multiplayer server and a binary protocol API for an non-existent app? the possibilities: not a lot hahahahaha. it's not like there is a lot to be invented.

all of this, to go back to my neocities Website and give up trying to host my own Web server LMAO

end of braindump.

[Gimmick]

Practically speaking, if your web server is too slow, then either scale horizontally or vertically. Most of the time you don't need to rewrite your code to use a different web server because people smarter than you and I long figured out how to do it the "best" way already.

But who gives a fuck about that! Pragmatism stifles creativity; if you want to create your own web server entirely using a custom userland TCP/IP stack, then go nuts! It's how we got TempleOS, SerenityOS and other such stuff that'll in all reality never see practical use, but are worth appreciating in their own right, after all.

[detergent1]

I started something

I went directly with libuv. I'm linking llhttp downloaded straight from nodeJS Git repo to handle HTTP 1.x stuff

I got a text/plain Hello, world! and the example llhttp parse working

libuv documentation is very lacking, but its developers are more interested to maintain it for nodeJS, thus I completely understand the idea of studying its internals because using it directly is indeed very rare

I'm currently working in a custom parser for a troff-like documents with semantic commands to generate HTML

I also found this: https://workers.cloudflare.com/

it says "Write in JS, Rust, C, and C++" and then "The first 100,000 requests each day are free"

so maybe I can adapt the code to run inside CloudFlare? pretty epic

[detergent1]

I progressed.

[Ferdian244]

i did try something like this back then, i was really struggling with libuv. im not a smart dev. then i found libhv that have similar purpose but much easier to understand for my small brain. ive been using that library everytime i need any networking stuff

[detergent1]

At 5/21/24 01:27 AM, Ferdian244 wrote:i did try something like this back then, i was really struggling with libuv. im not a smart dev. then i found libhv that have similar purpose but much easier to understand for my small brain. ive been using that library everytime i need any networking stuff

cool! I saw libhv mentioned here https://github.com/oz123/awesome-c?tab=readme-ov-file#networking-and-internet

it lists some other interesting servers like lwan, Kore and civetweb, but I was like "Node.js goes brrrrrr" and chose libuv

[detergent1]

and now... it is running http://191.252.220.165

coming soon: a domain name? or routing?

edit: oops, it already ran into an assertion and died lol

[detergent1]

At 5/22/24 10:20 PM, detergent1 wrote:and now... it is running http://191.252.220.165
coming soon: a domain name? or routing?

edit: oops, it already ran into an assertion and died lol

I think it won't crash anymore. annoying nonsensical http parsing.

[detergent1]

I think it is back, let's see how long it lasts now

http://191.252.220.165/movies

it currently has only one page, and none other. yes, I implemented routing!

I spent some time learning Ragel, studying HTTP 1.1 and replacing llhttp with a parser written by myself in Ragel. in the meanwhile, I found another HTTP parser called picohttpparser. both sucks hard, but I digress.

my HTTP implementation is not standards compliant: it answers before parsing messages fully, it does not recognize Content-Length and Transfer-Encoding, and it doesn't have graceful shutdown. graceful shutdown is an obscure topic, it will take a lot of time studying existing implementations to get it right.

also, my hot-reloading has never worked at all. Linux is incapable of watching file creation, which is a super common operation! I will have to watch the entire directory, but I think it'll be easier to write a new kernel module or something to implement this obvious feature, that of course Windows has had since ever.

I also noticed HTTP is fundamentally flawed for the following reason: it informs you the requested resource BEFORE informing the namespace, that is, the authority. the problem arises when a single server through a single IP serves many different websites and needs to use Host field to disambiguate. anyway, both HTTP protocol and HTTP parser libraries are very annoying.

also, there are no available HTTP conformance tests. someone said non-ironically to copy tests from other Web servers such as nginx and Apache! bollocks.

future directions... there is a lot to be done! extensive testing, testing HTTP conformance, improve logging, buy a domain name, add TLS, add more webpages, SEO, improve code, add compression such as brotli, etc.

[whiteingale]

At 12/30/23 04:50 AM, Gimmick wrote:Practically speaking, if your web server is too slow, then either scale horizontally or vertically. Most of the time you don't need to rewrite your code to use a different web server because people smarter than you and I long figured out how to do it the "best" way already.

that's something CIA would say.

I think this project is pretty cool idea and I want to look into machine coding with Assembly and the web development too. imagine the world where html was not and it was public assembly code on every site that'd be crazy ahahaha

[whiteingale]

At 4/16/24 02:54 PM, detergent1 wrote:

I also found this: https://workers.cloudflare.com/
it says "Write in JS, Rust, C, and C++" and then "The first 100,000 requests each day are free"
so maybe I can adapt the code to run inside CloudFlare? pretty epic

No, Cloudflare is pretty evil.

Do you need a server?

[detergent1]

At 8/16/24 10:36 AM, whiteingale wrote:

At 12/30/23 04:50 AM, Gimmick wrote:Practically speaking, if your web server is too slow, then either scale horizontally or vertically. Most of the time you don't need to rewrite your code to use a different web server because people smarter than you and I long figured out how to do it the "best" way already.

that's something CIA would say.

@Gimmick is correct

on the other hand, it saddens me that big projects rely on popularity to remain popular. big projects rarely explain what they are, what problems they solve, what are their shortcomings and the rationale for them, how they compare to their competitors or predecessors. then, a group of users are puzzled by temporary limitations, and lacking any information, they go on to mindlessly create alternatives instead of strengthening established solutions. this is my story. I'm confused with technology, so my only resort is to do it myself from scratch.

enter website for Apache HTTP server, it has zero information. enter nginx website, it overwhelms you with information, but it fails to explain how to answer text/plain "hello, you are our nth visitor". so does H20 website and CloudFlare. same drill for game engines.

I think this project is pretty cool idea and I want to look into machine coding with Assembly and the web development too. imagine the world where html was not and it was public assembly code on every site that'd be crazy ahahaha

I too imagine those things...

[detergent1]

At 8/16/24 10:38 AM, whiteingale wrote:

At 4/16/24 02:54 PM, detergent1 wrote:

I also found this: https://workers.cloudflare.com/
it says "Write in JS, Rust, C, and C++" and then "The first 100,000 requests each day are free"
so maybe I can adapt the code to run inside CloudFlare? pretty epic

No, Cloudflare is pretty evil.

source?

Do you need a server?

thanks, I already got a ridiculously cheap VPS :)