Glusterfs session #7 client, server interactions
0 likes65 views
- The document discusses client, server interactions in Glusterfs, including client connection, disconnection, and reconnection. - When a client connects, it first connects to glusterd to get the port for the brick, then connects to the brick to perform operations. On disconnection, resources are cleaned up and reconnection is attempted every 3 seconds. - During reconnection, the client tries to recover resources like open file descriptors by reopening them, though locks are not currently restored.
Glusterfs session #7 client, server interactions
- 1. Glusterfs Session #7 client, server interactions
- 2. Scope - Server init with all programs - Client Connection - interaction with server xlator - Client disconnection - interaction with server xlator - Client reconnection - interaction with server xlator
- 3. Server init - Server xlator sets up auth and programs it supports in the initialization of the translator - Demo server_init
- 4. Client connection - On Graph’s PARENT_UP notification rpc-clnt starts connecting to the brick. - Client first connects to glusterd on the node where the brick is located. - Once the connection is successful client asks glusterd for the port the brick is running on - Once it gets the port information, connection with glusterd is disconnected. - Connection to the brick is established with the port information received from glusterd
- 5. Client connection - continued - After successful connection with the brick - DUMP_OP rpc is sent to the brick to get the supported programs and their versions - Decides the version it will use for the fop rpcs v3 vs v4, newer clients go for v4 if the server has the support, otherwise v3. - HNDSHK_SETVOLUME rpc is sent now with the credentials and options the clients wants from the server - If server accepts the SETVOLUME, we have a successful connection on which glusterfs rpcs can be sent.
- 6. Client connection - continued - Server will setup the necessary structures(client_t) to make sure the resources can be cleaned up in case there is a disconnect. - Client_t data structure Code
- 7. Client disconnection - Client can disconnect because of the following reasons - Explicit disconnect either from client(graph switch) or server(auth changes) - Process dying - client/server Machine shutting off - Network issues - On client - all the open fds are marked bad, further fops on these fds would fail with EBADF. - Reconnection timers are started which would attempt reconnection.
- 8. Client disconnection - continued - On disconnect server will get a disconnect event and server_rpc_notify will - cleanup all the resources like locks, open fds. - All the ongoing requests will be errored while responding because the socket is disconnected - Structures corresponding to the connection will be cleaned up - Client_destroy, client_disconnect callbacks are called for each xlator - Code
- 9. Client reconnection - If client disconnected from server but the client process is still running, then client will keep trying to reconnect every 3 seconds. - Each connection attempt would be similar to the connection sequence. The only difference would be the client connection id will have the reconnection number incremented - Once the connection is restored client tries to recover the resources on the connection like open fds. At the moment locks are not restored.
- 10. Client reconnection - continued - For all the fds that are marked bad in client, a reopen request is sent to the server. This is best-effort. If reopens for any files/directories fail the fd would still be bad and would continue to give EBADF. - Once re-opens are completed the client xlator will notify the parents CHILD_UP event. New traffic on that xlator will start coming from this time onwards. - Code
- 11. Q & A