| def read_lines_from_file_as_data_chunks(file_name, chunk_size, callback, return_whole_chunk=False): | |
| """ | |
| read file line by line regardless of its size | |
| :param file_name: absolute path of file to read | |
| :param chunk_size: size of data to be read at at time | |
| :param callback: callback method, prototype ----> def callback(data, eof, file_name) | |
| :return: | |
| """ | |
| def read_in_chunks(file_obj, chunk_size=5000): |
| #!/usr/bin/env python | |
| from __future__ import print_function | |
| import json | |
| import logging | |
| from urllib2 import Request, urlopen, URLError, HTTPError | |
| from base64 import b64decode |
Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.
Terminology:
I wanted to be really able to explain to a fair amount of detail how does the program :command:`ls` actually work right from the moment you type the command name and hit ENTER. What goes on in user space and and in kernel space? This is my attempt and what I have learned so far on Linux (Fedora 19, 3.x kernel).
How does the shell find the location of 'ls' ?
| What exactly is "iowait"? | |
| To summarize it in one sentence, 'iowait' is the percentage | |
| of time the CPU is idle AND there is at least one I/O | |
| in progress. | |
| Each CPU can be in one of four states: user, sys, idle, iowait. | |
| Performance tools such as vmstat, iostat, sar, etc. print | |
| out these four states as a percentage. The sar tool can | |
| print out the states on a per CPU basis (-P flag) but most |
For this configuration you can use web server you like, i decided, because i work mostly with it to use nginx.
Generally, properly configured nginx can handle up to 400K to 500K requests per second (clustered), most what i saw is 50K to 80K (non-clustered) requests per second and 30% CPU load, course, this was 2 x Intel Xeon with HyperThreading enabled, but it can work without problem on slower machines.
You must understand that this config is used in testing environment and not in production so you will need to find a way to implement most of those features best possible for your servers.
Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a process.
| CODE | Meaning |
|---|---|
| D | Uninterruptible sleep (usually IO) |
| R | Running or runnable (on run queue) |
| S | Interruptible sleep (waiting for an event to complete) |
| T | Stopped, either by a job control signal or because it is being traced. |
| W | paging (not valid since the 2.6.xx kernel) |
| FROM madnight/docker-alpine-wkhtmltopdf as wkhtmltopdf_savior | |
| # STAGE for bundle & yarn install | |
| FROM ruby:2.4.3-alpine3.7 as builder | |
| ENV CA_CERTS_PATH /etc/ssl/certs/ | |
| ENV RAILS_ENV production | |
| ENV RAILS_LOG_TO_STDOUT true | |
| ENV RAILS_SERVE_STATIC_FILES true |
• 45-minute systems interview, focus on responding to real world problems with an unhealthy service, such as a web server or database. The interview will start off at a high level troubleshooting a likely scenario, dig deeper to find the cause and some possible solutions for it. The goal is to probe your knowledge of systems at scale and under load, so keep in mind the challenges of the Facebook environment.
• Focus on things such as tooling, memory management and unix process lifecycle.
More specifically, linux troubleshooting and debugging. Understanding things like memory, io, cpu, shell, memory etc. would be pretty helpful. Knowing how to actually write a unix shell would also be a good idea. What tools might you use to debug something? On another note, this interview will likely push your boundaries of what you know (and how to implement it).
Interview is all about taking an ambiguous question of how you might build a system and letting