Category: Cloud

It is a category with a aws, azure, gcp.. etc

04/15/2025

Azure Application Gateway 특징

Azure Application Gateway 에 특징이 있는데, 이에 대해 정리한다.

09/14/2024

CloudWatch Log 총 용량 계산하기

CloudWatch Log 의 총 용량의 사용량을 알아야 얼마 정도 비용이 나가는지를 추정할 수 있다. CloudWatch Log 화면에서 총 용량을 보여주면 고맙겠지만 그런게 없다보니 비용이 얼마정도 나가는지도 모르게된다.

Python 을 이용해서 총 용량을 계산해보자.

nextToken

AWS SDK 를 사용하는데 있어 항상 나오는 것이 nextToken 이다. AWS 는 무제한으로 API Call 을 하도록 허용하지도 않고 결과의 모두를 출력해주지 않는다. 다음을 보자.

client = session.client('logs')
response = client.describe_log_groups(limit=50)

1 2	client = session.client('logs') response = client.describe_log_groups(limit=50)

describe_log_groups 메소드에 인자값으로 limit 가 있다. 이 값은 Default 값이 있다. 문제는 이 인자값을 50 이상을 지정할 수가 없다는 것이다. CloudWatch Log 개수가 50개 이상이라면 딱 50개 만 출력이되는데, 나머지를 출력하고 싶다면 nextToken 인자를 주면 된다.

client = session.client('logs')
response = client.describe_log_groups(limit=50, nextToken=nxtToken)

1 2	client = session.client('logs') response = client.describe_log_groups(limit=50, nextToken=nxtToken)

일종의 Pagenation 개념으로 다음 페이지를 넘기며 출력을 하도록 하면 전체 CloudWatch Log 를 가지고 올 수 있다.

storedBytes

CloudWatch Log 에 SDK 에 리턴값은 Dictionary 로 나온다. 여기에는 logGroups 키로 하는 리스트형태의 Dictionary 인데, 여기에는 logGroupName, storedBytes 이며 storedBytes 값을 다 더하면 된다.

for group in response['logGroups']:
    print(f'logGrupName: {group['logGroupName']}, size: {group['storedBytes']}')
    totalStoredBytes += group['storedBytes']
nxtToken=response['nextToken']

for group in response['logGroups']:

print(f'logGrupName: {group['logGroupName']}, size: {group['storedBytes']}')

totalStoredBytes += group['storedBytes']

nxtToken=response['nextToken']

이런식으로 totalStoredBytes 에 전체용량을 저장하고 출력하면 된다.

전체소스

전체 소스는 다음과 같다.

import boto3

region = 'ap-northeast-2'
session = boto3.session.Session(profile_name='systemv', region_name=region)

client = session.client('logs')

nxtToken="first"
totalStoredBytes=0
while nxtToken:
    if nxtToken=="first":
        response = client.describe_log_groups(limit=50)
    else:
        response = client.describe_log_groups(limit=50, nextToken=nxtToken)
    try:
        for group in response['logGroups']:
            print(f'logGrupName: {group['logGroupName']}, size: {group['storedBytes']}')
            totalStoredBytes += group['storedBytes']
        nxtToken=response['nextToken']
    except KeyError:
        break

print(f'Total Bytes: {totalStoredBytes}')

import boto3

region = 'ap-northeast-2'

session = boto3.session.Session(profile_name='systemv', region_name=region)

client = session.client('logs')

nxtToken="first"

totalStoredBytes=0

while nxtToken:

if nxtToken=="first":

response = client.describe_log_groups(limit=50)

else:

response = client.describe_log_groups(limit=50, nextToken=nxtToken)

try:

for group in response['logGroups']:

print(f'logGrupName: {group['logGroupName']}, size: {group['storedBytes']}')

totalStoredBytes += group['storedBytes']

nxtToken=response['nextToken']

except KeyError:

break

print(f'Total Bytes: {totalStoredBytes}')

좀 더 개선된 코드를 생각해 볼 수 있다. 그건 여러분이들 해보라.

08/14/2024

AWS Sam with Golang

AWS Lambda 를 작성하는 방법, 정확하게는 Serverless 의 API 를 작성하는 방법으로 SAM 을 사용한다. 그런데, SAM 을 이용해서 Runtime 을 Golang 으로 하려고 할 경우에 잘 안되는 경우가 발생한다. 2023년 말에 변화가 있었다.

Migrating AWS Lambda functions from the Go1.x runtime to the custom runtime on Amazon Linux 2

이 문서는 위 문서의 내용을 요약한 것이다.

Error: There are no Template options available to be selected.

SAM 명령어를 이용해 Golang 런타임으로 생성할려고 하면 다음과 같이 오류가 발생한다.

]$ sam init --runtime go1.x
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1
Error: There are no Template options available to be selected.

]$ sam init --runtime go1.x

Which template source would you like to use?

1 - AWS Quick Start Templates

2 - Custom Template Location

Choice: 1

Error: There are no Template options available to be selected.

SAM 명령어에 옵션으로 –runtime go1.x 를 지정하면 위와같은 오류가 발생한다. AWS 공식문서에 따르면 이제 go1.x 는 없어졌다. 대신에 provided.al2 를 사용해야 한다.

]$ sam init --runtime provided.al2
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1

Choose an AWS Quick Start application template
        1 - Hello World Example
        2 - Infrastructure event management
        3 - Multi-step workflow
        4 - Lambda Response Streaming
        5 - DynamoDB Example
Template: 1

Which runtime would you like to use?
        1 - go (provided.al2)
        2 - graalvm.java11 (provided.al2)
        3 - graalvm.java17 (provided.al2)
        4 - rust (provided.al2)
Runtime: 1

]$ sam init --runtime provided.al2

Which template source would you like to use?

1 - AWS Quick Start Templates

2 - Custom Template Location

Choice: 1

Choose an AWS Quick Start application template

1 - Hello World Example

2 - Infrastructure event management

3 - Multi-step workflow

4 - Lambda Response Streaming

5 - DynamoDB Example

Template: 1

Which runtime would you like to use?

1 - go (provided.al2)

2 - graalvm.java11 (provided.al2)

3 - graalvm.java17 (provided.al2)

4 - rust (provided.al2)

Runtime: 1

Template.yml 파일 변화

–runtime 옵션에 provided.al2 를 사용하면 template.yml 파일에도 변화가 생긴다.

   Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: hello-world/
      Handler: bootstrap
      Runtime: provided.al2

Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction

Metadata:

BuildMethod: go1.x

Properties:

CodeUri: hello-world/

Handler: bootstrap

Runtime: provided.al2

Metadata 가 추가 된다. Runtime 이야 옵션으로 제공한 그대로 인데, Handler 가 bootstrap 이다. 이것을 바꿔서는 안된다. 또, Runtime 을 provided.al2 로 할 경우에 Lambda 의 Go 바이너리 실행 파일의 이름은 반드시 bootstrap 이여야 한다.

실행파일 이름은 반드시 bootstrap

Golang 의 람다 실행파일은 반드시 bootstrap 이어야 한다. 따라서 다음과 같이 컴파일해야 한다.

]$ GOARCH=amd64 GOOS=linux go build -o bootstrap main.go

1	]$ GOARCH=amd64 GOOS=linux go build -o bootstrap main.go

bootstrap 이 아닌 경우에는 실행되지 않는다.

-tags lambda.norpc 빌드 옵션 추가

bootstrap 바이너리를 빌드할때에는 반드시 -tags lambda.norpc 를 주어야 한다. AWS 공식 문서에 따르면 go1.x 에서는 필요했지만 provided.al2 로 런타임이 변경되면서 rpc 가 필요 없어졌다고 한다. 따라서 빌드할때에는 반드시 norpc 가 되도록 옵션을 지정해 줘야 한다.

]$ GOARCH=amd64 GOOS=linux go build -tags lambda.norpc -o bootstrap main.go

1	]$ GOARCH=amd64 GOOS=linux go build -tags lambda.norpc -o bootstrap main.go

변경사항이 많지는 않지만 위에 서술한 옵션들을 지키지 않을 경우에 문제가 된다. 실행파일 이름과 norpc 는 반드시 해줘야 하며 template.yml 에서도 필요한 내용이 있어야 한다. 그렇지 않으면 별것도 아닌 것으로 인해서 많은 시간을 허비하게 될 것이다.

04/18/2024

sam build 시 SSL 접속 에러 해결 방

다음과 같이 sam build 시 SSL 접속 에러가 발생할 수 있다.

]$ sam build --template template.yaml
Starting Build use cache
Manifest file is changed (new hash: 3298f13049d19cffaa37ca931dd4d421) or dependency folder (.aws-sam\deps\53036498-ac29-43da-b665-6ca82b0cd06d) is missing for (HelloWorldFunction), downloading
dependencies and copying/building source
Building codeuri: C:\labs\aws\Source_Code\lambda\make-thumnail\hello_world runtime: python3.12 metadata: {} architecture: x86_64 functions: HelloWorldFunction
 Running PythonPipBuilder:CleanUp
 Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
ERROR: Exception:
Traceback (most recent call last):
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\util\ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Python312\Lib\ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Python312\Lib\ssl.py", line 1046, in _create
    self.do_handshake()
  File "c:\Python312\Lib\ssl.py", line 1321, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 798, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\labs\aws\Lib\site-packages\pip\_internal\cli\base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\cli\req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\commands\download.py", line 132, in run
    requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 155, in __bool__
    return any(self)
           ^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 47, in _iter_built
    candidate = func()
                ^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 182, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 290, in __init__
    super().__init__(
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 222, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 301, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 519, in prepare_linked_requirement
    metadata_dist = self._fetch_metadata_only(req)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 371, in _fetch_metadata_only
    return self._fetch_metadata_using_link_data_attr(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 391, in _fetch_metadata_using_link_data_attr
    metadata_file = get_http_url(
                    ^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 109, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\network\download.py", line 134, in __call__
    resp = _http_get_download(self._session, link)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\network\download.py", line 117, in _http_get_download
    resp = session.get(target_url, headers=HEADERS, stream=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_internal\network\session.py", line 520, in request
    return super().request(method, url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 76, in send
    resp = super().send(request, stream, timeout, verify, cert, proxies)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\adapters.py", line 517, in send
    raise SSLError(e, request=request)
pip._vendor.requests.exceptions.SSLError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in
certificate chain (_ssl.c:1000)')))

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

]$ sam build --template template.yaml

Starting Build use cache

Manifest file is changed (new hash: 3298f13049d19cffaa37ca931dd4d421) or dependency folder (.aws-sam\deps\53036498-ac29-43da-b665-6ca82b0cd06d) is missing for (HelloWorldFunction), downloading

dependencies and copying/building source

Building codeuri: C:\labs\aws\Source_Code\lambda\make-thumnail\hello_world runtime: python3.12 metadata: {} architecture: x86_64 functions: HelloWorldFunction

Running PythonPipBuilder:CleanUp

Running PythonPipBuilder:ResolveDependencies

Build Failed

Error: PythonPipBuilder:ResolveDependencies - WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.

Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.

To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata

WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata

WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata

WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata

ERROR: Exception:

Traceback (most recent call last):

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 714, in urlopen

httplib_response = self._make_request(

^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 403, in _make_request

self._validate_conn(conn)

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 1053, in _validate_conn

conn.connect()

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connection.py", line 419, in connect

self.sock = ssl_wrap_socket(

^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\util\ssl_.py", line 449, in ssl_wrap_socket

ssl_sock = _ssl_wrap_socket_impl(

^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl

return ssl_context.wrap_socket(sock, server_hostname=server_hostname)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\Python312\Lib\ssl.py", line 455, in wrap_socket

return self.sslsocket_class._create(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\Python312\Lib\ssl.py", line 1046, in _create

self.do_handshake()

File "c:\Python312\Lib\ssl.py", line 1321, in do_handshake

self._sslobj.do_handshake()

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\adapters.py", line 486, in send

resp = conn.urlopen(

^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 826, in urlopen

return self.urlopen(

^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 826, in urlopen

return self.urlopen(

^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 826, in urlopen

return self.urlopen(

^^^^^^^^^^^^^

[Previous line repeated 2 more times]

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 798, in urlopen

retries = retries.increment(

^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\urllib3\util\retry.py", line 592, in increment

raise MaxRetryError(_pool, url, error or ResponseError(cause))

pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "c:\labs\aws\Lib\site-packages\pip\_internal\cli\base_command.py", line 180, in exc_logging_wrapper

status = run_func(*args)

^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\cli\req_command.py", line 245, in wrapper

return func(self, options, args)

^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\commands\download.py", line 132, in run

requirement_set = resolver.resolve(reqs, check_supported_wheels=True)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 95, in resolve

result = self._result = resolver.resolve(

^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 546, in resolve

state = resolution.resolve(requirements, max_rounds=max_rounds)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 397, in resolve

self._add_to_criteria(self.state.criteria, r, parent=None)

File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria

if not criterion.candidates:

^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\resolvelib\structs.py", line 156, in __bool__

return bool(self._sequence)

^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 155, in __bool__

return any(self)

^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 143, in <genexpr>

return (c for c in iterator if id(c) not in self._incompatible_ids)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 47, in _iter_built

candidate = func()

^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 182, in _make_candidate_from_link

base: Optional[BaseCandidate] = self._make_base_candidate_from_link(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 228, in _make_base_candidate_from_link

self._link_candidate_cache[link] = LinkCandidate(

^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 290, in __init__

super().__init__(

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 156, in __init__

self.dist = self._prepare()

^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 222, in _prepare

dist = self._prepare_distribution()

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 301, in _prepare_distribution

return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 519, in prepare_linked_requirement

metadata_dist = self._fetch_metadata_only(req)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 371, in _fetch_metadata_only

return self._fetch_metadata_using_link_data_attr(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 391, in _fetch_metadata_using_link_data_attr

metadata_file = get_http_url(

^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\operations\prepare.py", line 109, in get_http_url

from_path, content_type = download(link, temp_dir.path)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\network\download.py", line 134, in __call__

resp = _http_get_download(self._session, link)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\network\download.py", line 117, in _http_get_download

resp = session.get(target_url, headers=HEADERS, stream=True)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\sessions.py", line 602, in get

return self.request("GET", url, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_internal\network\session.py", line 520, in request

return super().request(method, url, *args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\sessions.py", line 589, in request

resp = self.send(prep, **send_kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\sessions.py", line 703, in send

r = adapter.send(request, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 76, in send

resp = super().send(request, stream, timeout, verify, cert, proxies)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\labs\aws\Lib\site-packages\pip\_vendor\requests\adapters.py", line 517, in send

raise SSLError(e, request=request)

pip._vendor.requests.exceptions.SSLError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in

certificate chain (_ssl.c:1000)')))

이럴때에는 실행하는 위치에 pip.conf 파일을 다음과 같은 내용으로 만든다.

[global]
trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org
               github.com

[global]

trusted-host = pypi.python.org

pypi.org

files.pythonhosted.org

github.com

그리고 이 파일에 대한 환경변수를 지정해 준다.

set PIP_CONFIG_FILE=pip.conf

1	set PIP_CONFIG_FILE=pip.conf

다시 sam build 를 수행하면 정상적으로 실행이 된다.

sam build --template template.yaml

1	sam build --template template.yaml

만일 컨테이너를 사용한다면 다음과 같이 하면 된다.

sam build --use-container --container-env-var PIP_CONFIG_FILE=pip.conf

1	sam build --use-container --container-env-var PIP_CONFIG_FILE=pip.conf

02/20/2024

CodeDeploy, Blue/Green 배포 실패시 인스턴스 삭제.

CodeDeploy, Blue/Green 배포 실패시 Rollback 이 되는데, 이때 인스턴스는 삭제가 되지 않는다. 삭제를 하기위해서는 CloudWatch, SNS 를 통해서 별도의 작업을 해줘야 한다.

출처: If a blue/green deployment fails on AWS, it leaves dangling instances. How do I ensure that old instances are terminated?

02/20/2024

AWS ECS, Service 생성 오류 디버깅

AWS ECS 에서 Cluster 를 생성하고 난 후에 Service 를 생성해야 한다. AWS ECS 는 Cluster, Service 생성시에 CloudFormation 으로 생성하게 되는데, 문제는 오류가 발생했을때다.

“Circuit Breaker” 라고 되어 있어서 그냥 CloudFormation 이 실행하면서 예기치못한 오류가 발생한 것으로 착각할 수 있다. 하지만, 이것이 문제가 아닐 수도 있다.

Service 생성 Event 확인

CloudFormation 에서 “CREATE_FAILED” 라고 되어 있어도 AWS ECS 에 Service 에는 service 가 생성되어 있다. Service 를 클릭하고 들어가서 Events 탭을 클릭하면 생성이 진행되었던 과정을 볼 수 있다.

여기서 task 로 시작하는 해쉬값 링크를 볼 수 있다. 이것을 클릭하고 들어가본다.

이제 뭐가 문제인지를 확실하게 보인다. “pull image mainfest has bean retried 1 time(s)” 로 되어 있고 ECR 에서 이미지를 못가지고 오고 있음을 알 수 있다.

01/08/2024

AWS CloudFront 인증서는 ACM eu-east-1 에서 해야 한다

AWS CloudFront 인증서는 ACM eu-east-1 (N. Virginia) 에서 발급 받아야 한다. 왜냐하면 CloudFront 때문인데, CloudFront 의 메인 리즌이 eu-east-1 로 되어 있어서 다른 리즌에 발급받은 인증서는 CloudFront 에서 사용할 수 없게 된다.

Q: 어느 리전에서 ACM을 사용할 수 있나요?

AWS 서비스를 사용할 수 있는 현재 리전을 확인하려면 AWS 글로벌 인프라 페이지를 참조하세요. Amazon CloudFront에서 ACM 인증서를 사용하려면 미국 동부(버지니아 북부) 리전에서 ACM 인증서를 요청하거나 가져와야 합니다. CloudFront 배포와 연동되어 있는 이 리전의 ACM 인증서는 해당 배포에 대해 구성된 모든 지리적 위치에 배포됩니다.

12/26/2023

Terraform 을 위한 VSCode 세팅

VSCode 에서 Terraform 을 사용하기 위한 세팅은 다음과 같다.

필요한 Extensions

HashiCorp Terraform
IntelliCode
Prettier – Code formatter

위와같은 Extensions 들을 설치하면 Terraform 을 사용하기가 편하다.

사용자 설정

Terraform 을 위한 사용자 설정은 다음과 같다.

    "terraform.languageServer.path": "D:\\Terraform\\terraform.exe",
    "[terraform]": {
        "editor.formatOnSaveMode": "file",
        "editor.formatOnSave": true,
        "editor.defaultFormatter": "hashicorp.terraform",
        "editor.tabSize": 2, // optionally
    },
    "[terraform-vars]": {
        "editor.formatOnSave": true,
        "editor.defaultFormatter": "hashicorp.terraform",
        "editor.formatOnSaveMode": "file",
        "editor.tabSize": 2 // optionally
    }

"terraform.languageServer.path": "D:\\Terraform\\terraform.exe",

"[terraform]": {

"editor.formatOnSaveMode": "file",

"editor.formatOnSave": true,

"editor.defaultFormatter": "hashicorp.terraform",

"editor.tabSize": 2, // optionally

"[terraform-vars]": {

"editor.formatOnSave": true,

"editor.defaultFormatter": "hashicorp.terraform",

"editor.formatOnSaveMode": "file",

"editor.tabSize": 2 // optionally

}

위 설정은 Prettier – Code formatter 가 설치되어 있어야 한다.

12/18/2023

Amazon MSK 2-AZ

Kafka 를 보게 되면서 Amazon MSK 를 보고 있는데, 문서를 보기에는 운영 환경에서 3-AZ 를 권장하고 있다. 성능 문서를 보았을때에도 3-AZ 를 권장하고 있고 Kafka 의 일반적은 환경에서도 3개의 브로커를 사용하도록 하고 있다.

하지만 Amazon MSK 에서는 2-AZ 로도 얼마든지 가능하다고 한다. 2019년 10월 16일 문서에 다음과 같이 되어 있다.

You can now expand your Amazon MSK clusters and deploy new clusters across 2-AZs
The new capability to expand allows your Amazon MSK cluster to scale out as your business grows by dynamically increasing the number of brokers within a cluster. The use of 2-AZ clusters is a good option for customers who already run Apache Kafka brokers in two AZs or use a replication factor of 2 by default.

Replication Factor 를 2로 할 경우에 얼마든지 2-AZ 로 할 수 있다고 되어 있다.

2025 8월
일	월	화	수	목	금	토
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30
31