# smelog Logger factory based on Structlog https://www.structlog.org/ and Sentry https://docs.sentry.io/platforms/python/ READ ABOUT THE STRUCTLOG!!! ## How to use First, install this library via `pip` pip install smelog Create Logger configs properly. ``` @dataclass class SentryConfig(): dsn: str # Sentry DSN level: int = logging.WARNING # Sentry logging level integrations: List[Integration] = field(default_factory=list) # Sentry support a number of integrations, more on that here: https://docs.sentry.io/platforms/python/ sample_rate: float = 1.0 # Sentry sample rate. This should have relatively low value in Production environment debug: bool = False # See additional info provided by Sentry about its internal work @dataclass class LoggerConfig(): name: str # Name of the application version: str # Version of the application level: int # Logging level, can be lower than SentryConfig.level, so Sentry will keep warnings and errors, but CloudWatch logs will have more debug and info log records environment: str # Environment where the component is executed sentry: Optional[SentryConfig] = None # Sentry config described above. If you don't need Sentry - just omit it is_local: bool = False # For local development it is easier to read logs written as a raw text rather than JSON-ified asyncio: bool = False # Use it for async applications, so logging will not block the execution ``` Common approach ```{.python3} import boto3 from smelog.entities import LoggerConfig, SentryConfig from smelog.factory import LoggerFactory, SmeBoundLogger def get_secret_value(key: str, client: Any) -> Dict[str, str]: value = client.get_secret_value(SecretId=key)['SecretString'] return json.loads(value) def lambda_handler(..., is_local: bool = False) -> Dict[str, Any]: environment = os.environment['ENVIRONMENT] is_production = environment == EnvironmentEnum.PROD log_config = LoggerConfig( name=APP_NAME, version=VERSION, level=logging.INFO if is_production else logging.DEBUG, environment=environment, is_local=is_local, ) if not is_local: sentry_dsn = get_secret_value( env_config['SENTRY_SECRET_KEY'], boto3.client('secretsmanager') )[APP_NAME] log_config.sentry = SentryConfig( dsn=sentry_dsn, integrations=[AwsLambdaIntegration(timeout_warning=True)] ) logger = LoggerFactory(log_config).get_logger(APP_NAME) logger = logger.bind(request_id=context.aws_request_id) # enrich logger context with extra fields do_something(logger) def do_something(logger: SmeBoundLogger): logger.info('blah') ``` Logger is implemented as a drop-in replacement for the standard `logging.Logger` ```{.python3} logger.debug(...) logger.info(...) logger.warning(...) logger.error(...) logger.exception(...) logger.critical(...) ``` Integrations: AWS Lambda (see: https://docs.sentry.io/platforms/python/guides/aws-lambda/) ```{.python3} from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration ... LoggerFactory( LoggerConfig( name=APP_NAME, version=VERSION, level=logging.INFO if is_production else logging.DEBUG, environment=environment, is_local=is_local, sentry=SentryConfig( dsn=sentry_dsn, integrations=[AwsLambdaIntegration(timeout_warning=True)] sample_rate=0.1 if is_production else 1.0, ) ) ).get_logger(APP_NAME) ``` Flask Application (see: https://docs.sentry.io/platforms/python/guides/flask/) ```{.python3} from flask import Flask from sentry_sdk.integrations.flask import FlaskIntegration ... LoggerFactory( LoggerConfig( name=APP_NAME, version=VERSION, level=logging.INFO if is_production else logging.DEBUG, environment=environment, is_local=is_local, sentry=SentryConfig( dsn=sentry_dsn, integrations=[FlaskIntegration()] sample_rate=0.1 if is_production else 1.0, ) ) ).configure() app = Flask(__name__) ``` If you're not sure which integration should you use just leave `integrations` field empty. Doesn't work? This library is open for contribution. ## Comparison sme-logger vs smelog sme-logger: ```{.python3} from sme_logger.logger import SimpleLoggerFactory config: Dict[str, Any] = { 'env': 'dev', } logger_factory = SimpleLoggerFactory(APP_NAME, environment != ENV_PROD, trace_id=trace_id) logger = logger_factory.get_logger() log_context: Dict[str, Any] = {} log_context.update(**config) logger.info('message: %s', 'something', extra=log_context) ``` smelog: ```{.python3} from smelog.entities import LoggerConfig, SentryConfig from smelog.factory import LoggerFactory config: Dict[str, Any] = { 'env': 'dev', } logger = LoggerFactory( LoggerConfig( name=APP_NAME, version=VERSION, level=logging.INFO if is_production else logging.DEBUG, environment=environment, is_local=False, ) ).get_logger(APP_NAME) log_context: Dict[str, Any] = {} log_context.update(**config) logger = logger.bind(trace_id=trace_id) logger = logger.bind(**log_context) # <-- now you don't need to pass log_context anywhere down the code logger.info('message: %s', 'something') ``` ## Using smelog with multiprocessing Key points here are: 1. to use bind_threadlocal() instead of bind() before splitting to several processes 2. set is_multiprocessing=True in LoggerConfig 3. call logger.close() in the end ```{.python3} from smelog.entities import LoggerConfig, SentryConfig from smelog.factory import LoggerFactory from structlog.threadlocal import bind_threadlocal def test_func(logger, service_name, numb): logger = logger.bind(service_name=service_name) <-- inside each process use bind() as usual logger.info(numb) return True IDS = [1, 2, 3, 4, ] config: Dict[str, Any] = { 'env': 'dev', } # init loger as usual logger = LoggerFactory( LoggerConfig( name=APP_NAME, version=VERSION, level=logging.INFO if is_production else logging.DEBUG, environment=environment, is_local=False, is_multiprocessing=True, ) ).get_logger(APP_NAME) log_context: Dict[str, Any] = {} log_context.update(**config) bind_threadlocal(**log_context) # <-- before splitting to several processes use bind_threadlocal try: with ProcessPoolExecutor(max_workers=4) as executor: fs = [executor.submit(test_func, logger, 'test_name', num) for num in numbers] for i, f in enumerate(as_completed(fs)): logger.info('iteration i: %s, progress: %s', i, f.result()) assert f.result() is True logger.info('message: %s', 'something') finally: logger.close() ``` ## Make commands - `make venv` - create local virtual env(base.pip dependencies) - `make devenv` - create local virtual env(dev.pip dependencies) - `make test` - run tests - `make lint` - run pylint check - `make clean` - clean unnecessary files - `make docker/login` - aws configuration need to be set first - `make docker/build` - build Docker images from a Dockerfile - `make deps/freeze` - freeze dependencies - `make deps/upgrade` - upgrade dependencies (when run `make test`, `make lint` command, all dependencies from requirements/test will be installed. When run `make venv` base.pip dependencies will be installed) ## Developers guide Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part of precommit hook. To register a new hook run: ```shell script pre-commit install ``` To run hooks without code committing: ```shell script pre-commit run ``` Sometimes these tools can have a conflict regarding import sorting. In this case consider to enable `include_trailing_comma` option at `setup.cfg` file. Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from wasting time at reading CI logs for failed build. In order to check everything is fine (and have a dry run for formatting and import sorting) consider following commands, e.g. for `smelog`: ```shell script cd smelog workon smelog # or './venv/bin/activate' isort --check-only -rc . && yapf --recursive --verbose --parallel --diff . \ && pylint --rcfile=pylintrc ./${PWD##*/} ./tests && mypy ${PWD##*/} && echo 'Exit code:' $? ``` It will run all checks inside a current directory.