2월 2015 - Voyager of Linux

Database, Redis

Redis 운영에 필요한 잡지식들.

02/28/2015

Redis 는 In memory 기반으로 동작하기 때문에 메모리 관리가 매우 중요한데, 먼저 리눅스에 메모리 관리를 Redis 운영에 적합하도록 설정하는 것이 좋다. 먼저 Redis 는 Swap 을 사용하지 않고 물리 메모리 내에서만 운영하는 환경이다.

vm.overcommit=2
vm.overcommit_ratio=99
vm.swappiness=0

echo never > /sys/kernel/mm/transparent_hugepage/enabled

vm.overcommit=2

vm.overcommit_ratio=99

vm.swappiness=0

echo never > /sys/kernel/mm/transparent_hugepage/enabled

가상 메모리 overcommit 에 대해서 2 로 설정하는 것이 좋다. 2로 설정하게 되면 리눅스가 사용가능한 메모리는 다음과 같이 계산한다.

swap + (physical memory * overcommit_ratio)

1	swap + (physical memory * overcommit_ratio)

예를들어 32GB 물리 메모리를 가지고 있고 overcommit_ratio 가 50%라면 16GB 물리 메모리만 사용하게 된다. 그래서 overcommit_ratio 를 99% 로 설정을하면 모든 메모리를 사용하게 된다. swappiness 는 […]

Database, Redis

Python redis 테스트 스크립트.

02/24/2015

랜덤으로 키를 생성해서 값을 넣도록 만든 Python redis 테스트 스크립트.

#!/usr/bin/env python

from multiprocessing import Process
import redis
import random
import string
import time

def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
    return ''.join(random.choice(chars) for x in range(size))

data = id_generator(12, "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()")

def g(idx):
        r_server = redis.Redis('localhost')
        for i in xrange(idx * 1024 * 1024 * 2, (idx+1)*1024*1024 * 2):
                key = 'PR%s'%(id_generator(random.randrange(6,15), "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()"))
                if random.randrange(1,100) > 70:
                        r_server.setex(key, id_generator(random.randrange(128,1024), "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()"),random.randrange(1,350))
                        print "expire key: %s" % key
                else:
                        r_server.set(key, id_generator(random.randrange(128,1024), "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()"))
                        print "permanent key: %s" % key
                time.sleep(0.2)

if __name__ == '__main__':

    pids = []
    for i in xrange(80):
        p = Process(target=g, args=(i,))
        pids.append(p)
        p.start()

    for pid in pids:
        pid.join()

#!/usr/bin/env python

from multiprocessing import Process

import redis

import random

import string

import time

def id_generator(size=6, chars=string.ascii_uppercase + string.digits):

return ''.join(random.choice(chars) for x in range(size))

data = id_generator(12, "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()")

def g(idx):

r_server = redis.Redis('localhost')

for i in xrange(idx * 1024 * 1024 * 2, (idx+1)*1024*1024 * 2):

key = 'PR%s'%(id_generator(random.randrange(6,15), "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()"))

if random.randrange(1,100) > 70:

r_server.setex(key, id_generator(random.randrange(128,1024), "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()"),random.randrange(1,350))

print "expire key: %s" % key

else:

r_server.set(key, id_generator(random.randrange(128,1024), "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz!@#$%^&*()"))

print "permanent key: %s" % key

time.sleep(0.2)

if __name__ == '__main__':

pids = []

for i in xrange(80):

p = Process(target=g, args=(i,))

pids.append(p)

p.start()

for pid in pids:

pid.join()

Java, Programming

tomcat jmx 소스

02/11/2015

JMX 소스

import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.openmbean.CompositeData;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

import java.io.Console;

public class check_jmx {

	public static final char ESC = 27;
	
	public static void main(String[] args) {
		// TODO Auto-generated method stub
		try {
			// Create an RMI connector client and
			// connect it to the RMI connector server
			//
			JMXServiceURL url =
				new JMXServiceURL("service:jmx:rmi:///jndi/rmi://127.0.0.1:9999/jmxrmi");
			JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
			MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
			
			ObjectName tomcatThreadPoolObh = new ObjectName("Catalina:type=Executor,name=tomcatThreadPool");
			Object activeCnt = mbsc.getAttribute(tomcatThreadPoolObh, "activeCount");
			
			ObjectName memoryObh = new ObjectName("java.lang:type=Memory");
			CompositeData memoryRes = (CompositeData) mbsc.getAttribute(memoryObh, "HeapMemoryUsage");
			
			Console c = System.console();
			if (c == null) {
				System.err.println("no console");
				System.exit(1);
			}
			
			// clear screen only the first time
			c.writer().print(ESC + "[2J");
			c.flush();
			Thread.sleep(200);
		
			for (int i = 0; i < 100000; ++i) {
				// reposition the cursor to 1|1
				c.writer().print(ESC + "[1;1H");
				c.flush();
				
				//c.writer().println("hello " + i);
				c.writer().println("activeCount = " + activeCnt);
				c.writer().println("HeapMemoryUsage committed = " + memoryRes.get("committed"));
				c.writer().println("HeapMemoryUsage init = " + memoryRes.get("init"));
				c.writer().println("HeapMemoryUsage max = " + memoryRes.get("max"));
				c.writer().println("HeapMemoryUsage used = " + memoryRes.get("used"));
				c.flush();
			
				Thread.sleep(5000);
			}
			
			/*
			echo("activeCount = " + activeCnt);
			echo("HeapMemoryUsage committed = " + memoryRes.get("committed"));
			echo("HeapMemoryUsage init = " + memoryRes.get("init"));
			echo("HeapMemoryUsage max = " + memoryRes.get("max"));
			echo("HeapMemoryUsage used = " + memoryRes.get("used"));
			*/

			jmxc.close();
			
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	private static void echo(String msg) {
		System.out.println(msg);
	}

}

import javax.management.MBeanServerConnection;

import javax.management.ObjectName;

import javax.management.openmbean.CompositeData;

import javax.management.remote.JMXConnector;

import javax.management.remote.JMXConnectorFactory;

import javax.management.remote.JMXServiceURL;

import java.io.Console;

public class check_jmx {

public static final char ESC = 27;

public static void main(String[] args) {

// TODO Auto-generated method stub

try {

// Create an RMI connector client and

// connect it to the RMI connector server

JMXServiceURL url =

new JMXServiceURL("service:jmx:rmi:///jndi/rmi://127.0.0.1:9999/jmxrmi");

JMXConnector jmxc = JMXConnectorFactory.connect(url, null);

MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();

ObjectName tomcatThreadPoolObh = new ObjectName("Catalina:type=Executor,name=tomcatThreadPool");

Object activeCnt = mbsc.getAttribute(tomcatThreadPoolObh, "activeCount");

ObjectName memoryObh = new ObjectName("java.lang:type=Memory");

CompositeData memoryRes = (CompositeData) mbsc.getAttribute(memoryObh, "HeapMemoryUsage");

Console c = System.console();

if (c == null) {

System.err.println("no console");

System.exit(1);

}

// clear screen only the first time

c.writer().print(ESC + "[2J");

c.flush();

Thread.sleep(200);

for (int i = 0; i < 100000; ++i) {

// reposition the cursor to 1|1

c.writer().print(ESC + "[1;1H");

c.flush();

//c.writer().println("hello " + i);

c.writer().println("activeCount = " + activeCnt);

c.writer().println("HeapMemoryUsage committed = " + memoryRes.get("committed"));

c.writer().println("HeapMemoryUsage init = " + memoryRes.get("init"));

c.writer().println("HeapMemoryUsage max = " + memoryRes.get("max"));

c.writer().println("HeapMemoryUsage used = " + memoryRes.get("used"));

c.flush();

Thread.sleep(5000);

}

echo("activeCount = " + activeCnt);

echo("HeapMemoryUsage committed = " + memoryRes.get("committed"));

echo("HeapMemoryUsage init = " + memoryRes.get("init"));

echo("HeapMemoryUsage max = " + memoryRes.get("max"));

echo("HeapMemoryUsage used = " + memoryRes.get("used"));

jmxc.close();

} catch (Exception e) {

e.printStackTrace();

}

private static void echo(String msg) {

System.out.println(msg);

}

Database, Redis

Redis 설치하기

02/07/2015

Redis 는 Key-Value 로 데이터를 저장하는 NoSQL 서버 입니다. In Memory 기반으로 동작하기 때문에 빠른 응답속도를 보입니다. 이 문서는 Redis 설치에 관한 문서 입니다. 다운로드

wget http://download.redis.io/releases/redis-2.8.19.tar.gz

1	wget http://download.redis.io/releases/redis-2.8.19.tar.gz

의존성 패키지 설치 CentOS 의 경우에는 다음과 같이 설치를 해줍니다.

yum install gcc.x86_64 gcc-c++.x86_64 wget.x86_64 bzip2-devel.x86_64 pkgconfig.x86_64 openssl-devel.x86_64 make.x86_64 man.x86_64 nasm.x86_64 gmp.x86_64 gdbm-devel.x86_64 readline-devel.x86_64 compat-readline43.x86_64 ncurses-devel.x86_64 db4-devel.x86_64 automake* autoconf* pcre-devel.x86_64 -y

yum install tcl-devel.x86_64 tcl.x86_64

yum install gcc.x86_64 gcc-c++.x86_64 wget.x86_64 bzip2-devel.x86_64 pkgconfig.x86_64 openssl-devel.x86_64 make.x86_64 man.x86_64 nasm.x86_64 gmp.x86_64 gdbm-devel.x86_64 readline-devel.x86_64 compat-readline43.x86_64 ncurses-devel.x86_64 db4-devel.x86_64 automake* autoconf* pcre-devel.x86_64 -y

yum install tcl-devel.x86_64 tcl.x86_64

Ubuntu 의 경우에는 다음과 같이 설치를 해줍니다.

sudo apt-get install build-essential

sudo apt-get install tcl8.5

sudo apt-get install build-essential

sudo apt-get install tcl8.5

컴파일

make

make

설치 설치는 PREFIX 를 인자로 줘서 설치 디렉토리 위치를 지정해줄 수 있습니다.

make install PREFIX=/usr/local/redis

1	make install PREFIX=/usr/local/redis

Redis 의 설치는 오로지 실행파일들만 복사해 줍니다. 설정 Redis 는 설치 후에 각종 설정을 할 수 있도록 스크립트를 지원 […]

HowTo, Linux

Cassandra 데이터 모델.

02/03/2015