Abstract

Global routing has been a historically challenging problem in the electronic circuit design, where the challenge is to connect a large and arbitrary number of circuit components with wires without violating the design rules for the printed circuit boards or integrated circuits. Similar routing problems also exist in the design of complex hydraulic systems, pipe systems, and logistic networks. Existing solutions typically consist of greedy algorithms and hard-coded heuristics. As such, existing approaches suffer from a lack of model flexibility and usually fail to solve sub-problems conjointly. As an alternative approach, this work presents a deep reinforcement learning method for solving the global routing problem in a simulated environment. At the heart of the proposed method is deep reinforcement learning that enables an agent to produce a policy for routing based on the variety of problems, and it is presented with leveraging the conjoint optimization mechanism of deep reinforcement learning. Conjoint optimization mechanism is explained and demonstrated in detail; the best network structure and the parameters of the learned model are explored. Based on the fine-tuned model, routing solutions and rewards are presented and analyzed. The results indicate that the approach can outperform the benchmark method of a sequential A* method, suggesting a promising potential for deep reinforcement learning for global routing and other routing or path planning problems in general. Another major contribution of this work is the development of a global routing problem sets generator with the ability to generate parameterized global routing problem sets with different size and constraints, enabling evaluation of different routing algorithms and the generation of training datasets for future data-driven routing approaches.

References

1.
Kahng
,
A. B.
,
2018
, “
Machine Learning Applications in Physical Design: Recent Results and Directions
,”
Proceedings of the 2018 International Symposium on Physical Design
,
Seaside, CA
,
Mar. 25–28
,
ACM
, pp.
68
73
.
2.
Schaller
,
R. R.
,
1997
, “
Moore’s Law: Past, Present and Future
,”
IEEE Spectrum
,
34
(
6
), pp.
52
59
. 10.1109/6.591665
3.
Hu
,
J.
, and
Sapatnekar
,
S. S.
,
2001
, “
A Survey on Multi-Net Global Routing for Integrated Circuits
,”
Integration
,
31
(
1
), pp.
1
49
. 10.1016/S0167-9260(01)00020-7
4.
Kramer
,
M. R.
,
1984
, “
The Complexity of Wirerouting and Finding Minimum Area Layouts for Arbitrary VLSI Circuits
,”
Adv. Comput. Res.
,
2
, pp.
129
146
.
5.
Chambon
,
R.
, and
Tollenaere
,
M.
,
1991
, “
Automated Al-Based Mechanical Design of Hydraulic Manifold Blocks
,”
Comput. Aided Des.
,
23
(
3
), pp.
213
222
. 10.1016/0010-4485(91)90091-A
6.
Kang
,
S.-S.
,
Myung
,
S.
, and
Han
,
S.
,
1999
, “
A Design Expert System for Auto-Routing of Ship Pipes
,”
J. Ship Prod.
,
15
(
1
), pp.
1
9
.
7.
Christodoulou
,
S. E.
, and
Ellinas
,
G.
,
2010
, “
Pipe Routing Through Ant Colony Optimization
,”
J. Infrastruct. Syst.
,
16
(
2
), pp.
149
159
. 10.1061/(ASCE)1076-0342(2010)16:2(149)
8.
Grayman
,
W. M.
,
Clark
,
R. M.
, and
Males
,
R. M.
,
1988
, “
Modeling Distribution-System Water Quality; Dynamic Approach
,”
J. Water Res. Plan. Manage.
,
114
(
3
), pp.
295
312
. 10.1061/(ASCE)0733-9496(1988)114:3(295)
9.
Ehmke
,
J. F.
,
Steinert
,
A.
, and
Mattfeld
,
D. C.
,
2012
, “
Advanced Routing for City Logistics Service Providers Based on Time-Dependent Travel Times
,”
J. Comput. Sci.
,
3
(
4
), pp.
193
205
. 10.1016/j.jocs.2012.01.006
10.
Barceló
,
J.
,
Grzybowska
,
H.
, and
Pardo
,
S.
,
2007
, “
Vehicle Routing and Scheduling Models, Simulation and City Logistics
,”
Dynamic Fleet Management
,
Springer
,
New York
, pp.
163
195
.
11.
Lee
,
J.
,
Bose
,
N.
, and
Hwang
,
F.
,
1976
, “
Use of Steiner’s Problem in Suboptimal Routing in Rectilinear Metric
,”
IEEE Trans. Circuits Syst.
,
23
(
7
), pp.
470
476
. 10.1109/TCS.1976.1084243
12.
Moffitt
,
M. D.
,
2008
, “
Maizerouter: Engineering an Effective Global Router
,”
IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
,
27
(
11
), pp.
2017
2026
. 10.1109/TCAD.2008.2006082
13.
Kastner
,
R.
,
Bozorgzadeh
,
E.
,
Sarrafzadeh
,
M.
, and
Sarrafzadeh
,
M.
,
2000
, “
Predictable Routing
,”
Proceedings of the 2000 IEEE/ACM International Conference on Computer-Aided Design
,
San Jose, CA
,
Nov. 5–9
, New York, pp.
110
114
.
14.
Cho
,
M.
, and
Pan
,
D. Z.
,
2007
, “
Boxrouter: A New Global Router Based on Box Expansion and Progressive ILP
,”
IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
,
26
(
12
), pp.
2130
2143
. 10.1109/TCAD.2007.907003
15.
Hasan
,
N.
,
1987
, “
A Force-Directed Global Router
,” PhD thesis,
University of Illinois
,
Urbana-Champaign, IL
.
16.
Mnih
,
V.
,
Kavukcuoglu
,
K.
,
Silver
,
D.
,
Graves
,
A.
,
Antonoglou
,
I.
,
Wierstra
,
D.
, and
Riedmiller
,
M.
,
2013
, “
Playing Atari With Deep Reinforcement Learning
,” preprint arXiv:1312.5602.
17.
Mnih
,
V.
,
Kavukcuoglu
,
K.
,
Silver
,
D.
,
Rusu
,
A. A.
,
Veness
,
J.
,
Bellemare
,
M. G.
,
Graves
,
A.
,
Riedmiller
,
M.
,
Fidjeland
,
A. K.
,
Ostrovski
,
G.
,
Petersen
,
S.
,
Beattie
,
C.
,
Sadik
,
A.
,
Antonoglou
,
I.
,
King
,
H.
,
Kumaran
,
D.
,
Wierstra
,
D.
, and
Legg
,
S.
,
2015
, “
Human-Level Control Through Deep Reinforcement Learning
,”
Nature
,
518
(
7540
), p.
529
. 10.1038/nature14236
18.
Van Hasselt
,
H.
,
Guez
,
A.
, and
Silver
,
D.
,
2016
,
Deep Reinforcement Learning With Double Q-Learning
, Vol.
2
,
AAAI
,
Phoenix, AZ
, p.
5
.
19.
Kulkarni
,
T. D.
,
Narasimhan
,
K.
,
Saeedi
,
A.
, and
Tenenbaum
,
J.
,
2016
, “
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
,”
Advances in Neural Information Processing Systems
, pp.
3675
3683
.
20.
Horgan
,
D.
,
Quan
,
J.
,
Budden
,
D.
,
Barth-Maron
,
G.
,
Hessel
,
M.
,
Van Hasselt
,
H.
, and
Silver
,
D.
,
2018
, “
Distributed Prioritized Experience Replay
,” preprint arXiv:1803.00933.
21.
Silver
,
D.
,
Hubert
,
T.
,
Schrittwieser
,
J.
,
Antonoglou
,
I.
,
Lai
,
M.
,
Guez
,
A.
,
Lanctot
,
M.
,
Sifre
,
L.
,
Kumaran
,
D.
,
Graepel
,
T.
,
Lillicrap
,
T.
,
Simonyan
,
K.
, and
Hassabis
,
D.
,
2018
, “
A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play
,”
Science
,
362
(
6419
), pp.
1140
1144
. 10.1126/science.aar6404
22.
Bellman
,
R.
,
1957
, “
A Markovian Decision Process
,”
Indiana Univ. Math. J.
,
6
, pp.
679
684
. 10.1512/iumj.1957.6.56038
23.
Sechen
,
C.
,
2012
,
VLSI Placement and Global Routing Using Simulated Annealing
, Vol.
54
,
Springer Science & Business Media
,
New York
.
24.
de Vincente
,
J.
,
Lanchares
,
J.
, and
Hermida
,
R.
,
1998
, “
RSR: A New Rectilinear Steiner Minimum Tree Approximation for FPGA Placement and Global Routing
,”
Proceedings of the 24th Euromicro Conference
, PISA,
July 10–13
, Vol.
1
,
New York
, pp.
192
195
.
25.
Ho
,
J.-M.
,
Vijayan
,
G.
, and
Wong
,
C.-K.
,
1990
, “
New Algorithms for the Rectilinear Steiner Tree Problem
,”
IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
,
9
(
2
), pp.
185
193
. 10.1109/43.46785
26.
Wong
,
C. C. M. S. C.
,
1990
, “
Global Routing Based on Steiner Min-Max Trees
,”
IEEE Trans. Comput. Aided Des.
,
9
, pp.
1315
1325
.
27.
Cho
,
M.
,
Lu
,
K.
,
Yuan
,
K.
, and
Pan
,
D. Z.
,
2007
, “
Boxrouter 2.0: Architecture and Implementation of a Hybrid and Robust Global Router
,”
Proceedings of the 2007 IEEE/ACM International Conference on Computer-Aided Design
,
San Jose, CA
,
Nov. 4–8
, New York, pp.
503
508
.
28.
Qi
,
Z.
,
Cai
,
Y.
, and
Zhou
,
Q.
,
2014
, “
Accurate Prediction of Detailed Routing Congestion Using Supervised Data Learning
,”
2014 IEEE 32nd International Conference on Computer Design (ICCD)
,
Seoul, Korea
,
Oct. 19–22
, New York, pp.
97
103
.
29.
Zhou
,
Q.
,
Wang
,
X.
,
Qi
,
Z.
,
Chen
,
Z.
,
Zhou
,
Q.
, and
Cai
,
Y.
,
2015
, “
An Accurate Detailed Routing Routability Prediction Model in Placement
,”
2015 6th Asia Symposium on Quality Electronic Design (ASQED)
,
Penang, Malaysia
,
Aug. 3–5
,
IEEE
, pp.
119
122
.
30.
Sutton
,
R. S.
, and
Barto
,
A. G.
,
1998
,
Introduction to Reinforcement Learning
, Vol.
135
,
MIT Press
,
Cambridge
.
31.
Nair
,
V.
, and
Hinton
,
G. E.
,
2010
, “
Rectified Linear Units Improve Restricted Boltzmann Machines
,”
Proceedings of the 27th International Conference on Machine Learning (ICML-10)
,
Haifa, Israel
,
June 21–24
, pp.
807
814
.
32.
Nguyen
,
T. T.
,
Nguyen
,
N. D.
, and
Nahavandi
,
S.
,
2018
, “
Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
,” CoRR, abs/1812.11794.
33.
Hu
,
J.
, and
Wellman
,
M. P.
,
1998
, “
Multiagent Reinforcement Learning: Theoretical Framework and an Algorithm
,”
ICML
,
Madison, WI
,
July 24–27
, pp.
242
250
.
34.
Buşonic
,
L.
,
Babuška
,
R.
, and
De Schutter
,
B.
,
2008
, “
A Comprehensive Survey of Multiagent Reinforcement Learning
,”
IEEE Trans. Syst. Man Cybern. C (Appl. Rev.)
,
38
(
2
), pp.
156
172
. 10.1109/TSMCC.2007.913919
35.
Tan
,
M.
,
1993
, “
Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents
,”
Proceedings of the Tenth International Conference on Machine Learning
,
Amherst, MA
,
July 27–29
, pp.
330
337
.
36.
Tang
,
Y.
,
Agrawal
,
S.
, and
Faenza
,
Y.
,
2019
, “
Reinforcement Learning for Integer Programming: Learning to Cut
,” preprint arXiv:1906.04859.
37.
Li
,
Z.
,
Chen
,
Q.
, and
Koltun
,
V.
,
2018
, “
Combinatorial Optimization With Graph Convolutional Networks and Guided Tree Search
,”
Advances in Neural Information Processing Systems
, pp.
539
548
.
38.
Gasse
,
M.
,
Chételat
,
D.
,
Ferroni
,
N.
,
Charlin
,
L.
, and
Lodi
,
A.
,
2019
, “
Exact Combinatorial Optimization With Graph Convolutional Neural Networks
,” preprint arXiv:1906.01629.
39.
Schaul
,
T.
,
Quan
,
J.
,
Antonoglou
,
I.
, and
Silver
,
D.
,
2015
, “
Prioritized Experience Replay
,” preprint arXiv:1511.05952.
You do not currently have access to this content.